For the first 10 years of my 'serious cycling life' I never had any cycling-specific clothing. I rode my old Fuji S-10s well over 40,000 miles over those years. Many Centuries. Many two-three-day rides, and even a 1000-mile 'tour' of 90-120-mile days. Again, just in street clothes.
Now I have a pair of padded/chamois cycling shorts that I'll wear on rides of over 40 miles (two hours) - and only under my short-inseam hiking shorts (pockets!). On rides of less than 40 miles, I wear normal clothes - hiking shorts and a hi-vis wicking t-shirt (short- or long-sleeve).