Since you don't care if anyone thinks you look like a dork, why come to a public forum and broadcast it?Seeing as you refer to "cyclists" in the third person, you apparently see yourself outside our community anyway.
Just keep flying your freak flag.
BTW, on long rides where I anticipate needing to remove lower layer, I wear shorts.