Granted, I just got back to logging substantial miles last year; but all last year and the off and on years before were ridden in regular old cargo shorts and a polo, or T shirt. Never experienced a honk, or a hand gesture, no discouraging words thrown my way. So I go out and buy myself a pair of cycling shorts and viola! 2 idiots on the same day.
Gotta be the spandex.