Lots of things that qualify as "regular street clothes" could also qualify as dorky so I'm not sure we should be judging.

Personally, like lots of fashion that's sort of on the edge of mainstream, I think it depends on the person wearing it and the context. A 20 something guy wearing knickers on a bike in a bike friendly US city wouldn't even be noticed. That same guy working for some Silicon Valley startup probably would not be remarkable even in an office setting.

Now take a 55 year old guy working at an insurance company in the most bike unfriendly suburb you can imagine. That guy is going to standout in a negative way.