I remember my buddies making fun of me for wearing my old cycling shoes for flying. My stated excuse was that the stiff soles helped in applying more direct pressure to the rudder pedals and especially the toe-brakes. My other excuse was that they're just so much more comfortable. And in case some of you were wondering, no, these shoes did not have cleats mounted to them.
As far as clothing goes... wear what's comfortable. I personally wouldn't wear lycra for just sitting around the house watching the TdF because to me, that's not comfortable. Riding shorts work well when you're sitting on a saddle but lounging about on the couch is a different story. As far as for being self-conscious... I couldn't care less. I was telling my wife that when I go out of town for our annual meeting and company picnic, that I'm going to hook up with a coworker who'll lend me his bike and we're going to ride from the office to the picnic area (20 miles each way). She asked me if I was going to wear my clown outfit and I told her yes. She said that might not be a good idea since my coworkers and boss will be there. I responded by telling her that it's mainly a social gathering and it's at a waterpark where some people will be in swimsuits. At anyrate, I'm not about to show up with saddle-sores and a drenched in sweat shirt just because my riding outfit is a little flashy.