You don't NEED to have cycling specific clothing, look at all the bike messengers riding around in Carharts rolled up to their knees all day long. I prefer Filsons, they dry faster.
All day rides, tights are nice, but bombing around town, anything goes. If you're thrifty, try goodwill for wool sweaters, athletic tights and the like. Oh, and if you're dropping $$$, Smartwool and Ibex wool clothing ROCKS! Ibex makes padded shorts, tights, and knickers as well as jerseys.