OP, I can help you feel a bit better.
Virtually all riders, from the age of 13 on have the same power and speed. The difference is the bike. All the guys you see that are riding much faster than you are on electric bikes. They have supplemental power which isn't the same as supernatural power. Some of these bikes are quite discreet. You can't see the battery or the supplemental motor built into the bottom bracket. A dirty untold secret is, the BB's of modern carbon bikes aren't ball shaped to increase stiffness...or rather its a by-product, but they are made more bulbous to house an electric motor. Likely the skinny guy is no stronger than you and perhaps weaker but has supplemental electrical power. Only thing you can do is buy a Specialized Turbo and remove the governor.
Same applies to the guys that are riding in the TdF. They are all on e-bikes and the great race has now turned into more of a pageant or ceremony and the win is now generally rewarded to the team with the best looking jerseys for TV.
HTH