Roadies can be like a bunch of teenage girls when it comes to cycling clothes, bikes, equipment, etc. It's all about having the gear perceived to enhance peak performance. However, a cyclist wearing baggy clothes and riding touring/commuter/cylocross bike can also be perceived as a threat to the safety of the group. Riding in packs at fast speeds can be dangerous, and someone not experienced in the nuances of pace-riding can easily cause a wreck that injures a lot of people and ruins equipment. If you show up at a roadie group ride dressed as you described, whether justified or not, many of the other riders may assume that you lack the skills and endurance to ride safely in a pack.
I no longer ride with the fast groups around town for those reasons and many others. Instead I ride with a group of friends that I know and trust, and who don't care if I show up on my commuter bike with fenders and large seat bag attached. If you look around, you can probably find a similar group.