Hello OP,
After I got married I decided to wear the gear. So I did that for a few years and now have gone back to whatever shorts and shirts suit me for riding best, that already are in my closets. Can't say I've reached a daily distance where I start to think about going back to bicycle specific riding clothes though. The further I ride, the better I feel (given the time allotted). So far, so far I haven't thought, hmm, I should go back to the riding clothing.
Some people like the feel of lycra or that kind of material when they ride. Makes them feel more... right. If you fit into that category, you should wear it and accept what works best for you.
Harv