I do not know what you mean by "average" speed. Some people seem to mean their usual "cruising" speed. A true average speed is different and is almost always about 2 mph lower than the cruising speed.
Another thing is it takes a fair bit of riding time for a cyclist to build up to be able to ride at a steady 17-18 mph pace. Even being very fit in other sports will not help much. There is not much transfer. Think of it this way. Micheal Jordan was a really good professional basketball player (an large understatement). But he was just another minor league baseball player. I have ridden with ex elite tennis players, hockey players, professional body builders and others. They were good but not great. As I said, the performance does not transfer from other sports to cycling and vice versa.
Size is not a huge disadvantage in cycling. Large cyclists often do well in being able to push high speeds on flat terrain (they tend to have big engines). Of course, being large really is not good for climbing hills. I often see light women climb hills very well.
So if you lost a pile of weight and looked like the proverbial 90 lb weakling from the waist up, you would almost certainly climb better. But unless you are a professional cyclist, why would you want to?