Just to start this off with background, I'm a 120 pound male recreational cyclist.
You have to remember that, unless the person is a pro cyclist with all her muscle in her legs, or his in my case, the amount of muscle is generally pretty proportional to that of a larger person. By this I mean that, off the bike, a fit 180 lb person and a fit 120 lb person will have roughly the same performance overall.
Now add a 20 pound bike. For the 180 lb person, this is an extra 1/9th of his weight. For a 120 lb person this is an extra 1/6th of his weight! Big difference, which makes lugging the bike up the hill that much harder for the small person.
Furthermore, the rule of thumb is that while volume increases by a factor of 3, surface area increases by a factor of 2. So while the 180 lb person has let's say 1.5 times the muscle of the 120 lb person, he also has only about 1.3 times the volume. Therefore, proportionally the bigger person has the advantage.
When it comes to pro level though, with cyclists having 90% of their whole mass located in 3 or 4 muscles, well, then it gets tricky...