The basic problem with spandex cycling clothes is that for men at least, regular clothes tend to be loose fitting with subdued colors and patterns.

Men often wear jerseys from other sports (basketball, baseball, football, etc) when out and about and while the patterns and colors may be bold, the jerseys themselves are often oversized.

Colorful and form fitting cycling clothes represent quite a departure from what men normally wear and to me that's what the ant-spandex viewpoint boils down to. That it's a sensible choice for certain types of cycling is beside the point.