I've yet to actually get a cycling jersey. Bright orange Under Armour for me. Though, I have been pondering one of those mystery bags from Velowear.com... I'm not picky.
I've seen all kinds of clothing on the local singletrack, though. Everything from topless to full kit. Really, as long as you can carry everything you need and you're comfortable in it, go for it.