I completely disagree with Chris Hoy. I am 62 in 6 weeks and have ridden 3000 miles so far this year. I wear bike shorts and full kit. Not flashy colors and team kit, but understated and comfortable, form fitting stuff.
I don't wear it because I look good. Most people I run into while riding underestimate my age by 10 years and more and are suprised to hear how old I am. I think I owe that partly to the fact that I do ride alot. I dont' see it. I see my physical flaws and know, realisticly, I don't look that good. But on a 40 or 50 mile day I want to feel good and make it to the end of the ride. So I wear bike specific gear.
However, I don't stop at coffee shops (I refuse to pay 5 bucks for coffee) or go shopping while riding. I have on occasion, stopped in a city park and taken a nap in the grass. I ride over to my folks assisted-living retirement home, filled with people older than me and get a large number of dubious compliments from older folks that most likely can't see straight or clearly anyway.
I guess what I'm trying to say is that bike specific clothing and "lycra" (my brothers call it "women's underwear") make the bike much more comfortable and Chris Hoy isn't going to change my mind.