i went the other way. used to always wear lycra and never questioned it
however, i like cycle-specific clothing (not lycra). the materials are fitted, strech and are cut for cycling and wearing normally off the bike
i've done serious miles in them and was comfortable

if you want mixed use clothing or just something "normal" looking, there are options.
i've found it allows me to cycle more and not think of it as something i have to change clothes every time i ride.

racing and training is best in lycra but there are options.