If you look at pix of professional cyclists, especially close ups of them riding at >20mph you will be struck by how skinny they are. Their upper body would be expected to be average in muscle development, about like a 14yr boy. But the legs, for most of them are NOT large, but the muscles are very well defined
From the front the legs look large but look again from the side and you will see they are not. Most notable example was the TdF mountain contender Michael Rasmussen, but all the TdF, Giro and Vuelta biggies had asthenic appearing legs. The fact that you can drop most of your peers on hills shows you have considerable basic power, what you need is interval training to improve your maximal aerobic capacity. More weight/muscle is not the answer.