Sport and utility overlap more the longer and faster you want to ride. I commute in spandex on drop bar road bikes, switching to casual clothes for very short utility rides.
In another thread I commented that the normalcy of cycling clothing depended on the typical nature of cycling in the locality. Localities with long distances between typical trip endpoints see more spandex clad commuters; in these areas the casually dressed cyclists stick out more, and invite presumptions of inability to drive or afford a car. In denser areas with short trip distances, casual clothes are more common among transportation cyclists of all socioeconomic status levels.
A well thought out transportation network that facilitates cycling that is enjoyable and efficient (including supportive of exercise-oriented cycling) will generate the optimum level of cycling in an affluent community. Under such conditions, cyclists will not be assumed to be indigent or kooks regardless of what they wear or ride.