I wear lycra and happy with it. Bike specific clothing is the best for well, cycling so why not wear it. Don't worry about what others say or think, the ones who may make sarky comments are often the ones way over weight and couch potatoes.
Anyway, at 61 I reckon I'm too old to be a MAMIL!