Wear what you want. If you're not comfortable in the spandex, then don't wear it.
Personally, I'm happiest in a loose fitting running shirt (thin and breathable) and either gel padded compression pants or Champion basketball shorts, depending how long the ride is.