No reason to avoid us, just be aware we are there, and on our particular rides we are probably moving a lot faster than you because our gears let it happen. If a roadie looks down on "recreational" riders, he/she is a turd. Period. I don't really care for dirt anymore, but at one time in my life, it was all I rode. A lot of my pals now were the same way, we just grew up, had families, and life overcame us, and we didn't have time or freedom to drive to a trail or hill or mountain anymore. I ride on the road, and I ride hard (because I like to ride hard)...but if I see someone on a MTB or hybrid or comfort bike, I give them notice I am passing, give them space, and don't give them a second thought, certainly not one of disparagement.
Engage some of us, you'll come to find out that most of us are no different than you, other than our preferred ride.