Being a guy who wears a full team kit for almost every type of riding (even commuting), I urge you to remember that cycling is as much about fantasizing as it is about riding for a lot of people. I ride a lot but like to fantasize that I am stronger and faster than I really am (hence the local team involvement). For most fat guys to fantasize about being faster than others, they need a likely target. So they often tend to pick a newbie looking guy with a less than sexy bike. This is a dangerous practice.
In my experience, it is the guy who shows up in the 1980's looking spandex with the clearly 1980's bike that you have to look out for. He is likely to be much faster than you might otherwise give him credit for. After all, the cahones it takes to ignore bicycling magazine's "Style Guy" should indicate a sense of self assurance that can possibly be backed up by some serious speed.
I maintain 5 bikes. 3 of them are sexy newer models that I could not really afford to replace. The other two are beaters selected for being ugly but in great mechanical shape. I have been suprised to discover that I will pull down a beater as often as a newer model for any given ride. My beaters are "real bikes" to be sure. They are dead on mechanically and totally dialed in for my fit. And, suprise, suprise, they are not significantly slower in ANY way compared to my 17lbs sexy machines (climbing and excelerating notwithstanding). I love my old stock as much if not more than my new bikes. There is no pleasure like skunking the insecure snobs on their $3,000 bikes as you blow their doors off on your $45 thrift store bike. And I mean a true butt-spanking....a random passing while out and about is not fair; you never know if they are on a recovery ride or something. But to be riding with them and smoke them....well, that is just plain fun!
My approach is that anyone who can regularly outride me is more than welcome to make fun of my gear. Get strong, get fast (sounds like you are well on your way, if not there already) and then challenge the next guy who gives you the "get a real bike" speach to let his riding do his talking. Most "coffee shop riders" who tend to do a lot of talking simply can't hang all that well at speed.