If you wanted to ride with a road group I don't think it would be a problem, but you'd have to find a more casual group. Road riding is fun even if you don't have all the gear, because of the drafting, taking turns to pull, etc. I eschewed a road bike (or even anything other than a MTB) for many years, partly because of cost and partly because of my perception of the road crowd, but now I ride road rides more often than others (excepting commuting).