AF,
There's nothing to be discouraged about. You aren't so unusual. I know at least one person in this forum whose been there
. And (fact of life) you will
always be able to go out and find people who can ride faster than you.
Not all the road warriors are "poseurs," and not all "poseurs" are true road warriors. I read a story about an avid cyclist in the bay area (Eric House) who basically blew away everyone in a century, riding in an oxford shirt. I guess he was making a statement.
Bottom line: pick your goals, move in that direction, and give it time to happen. When equipment (bikes, gear, etc.) become a significant limitation, upgrade. Ride bike and have fun.