Wearing your Lycra doesn't make you a roadie in disguise, it makes you a cross country rider. I would say that 60-70% of the riders I see on the trail here in CO are wearing Lycra, me included. Adding anything baggy just gets annoying when you are pedaling for half a day or more, especially on long climbs.

OP, if your concern is looking like a mountain biker, don't sweat it. Become an endurance rider.