I like mine to be tight for road riding so the sides don't flap in the wind. When I am trail riding I prefer a loose fitting jersey or a polyester workout shirt.
I have 3 slong sleeve jerseys, 6 or 7 short sleeve jerseys and 5 pair of spandex shorts.
I wear what feels good. It's all a personal preference and has no bearing on how well you can ride your bike. Just enjoy doing it your way.