When I resumed cycling in 2015 after many years away, I wore whatever shorts and t-shirts I already had. But as my fitness improved and my goals got tougher, I eventually gave in and resumed wearing form fitting kit on my road bike (not on my hybrids -- I still wear casual clothes for those rides).
Form fitting kit is safer, not just a little more aero. In spring we get some pretty fierce winds and loose fitting clothing can make it a chore just to hold a steady line in a ripping crosswind. Almost every ride I've taken the past month has been in 15-30+ mph wind, often with unpredictable gusts. Snug clothing gives the wind less to grab and use as a sail. It's reassuring on rural two-lane highways when drivers sometimes misjudge passes and don't quite give me enough distance. Being able to hold a steady line helps us all.
Not a big deal when I'm enjoying a relaxing 10-12 mph cruise on my hybrids. But on a downhill blast at 40 mph with a tailwind and crosswind gusts from passing vehicles, every little bit helps keep things steady.
The past couple of rides I wore jerseys that are somewhat form fitting, but not race fit or aero jerseys for time trials -- I have those and wear them often, but the weather suddenly heated up this weekend and my other slightly relaxed jerseys are cooler fabric. The race fit/aero jerseys use some slick, thick plasticky fabric around the shoulders that supposedly makes them slippery in the wind, but the fabric doesn't breathe as well.
I could feel a bit of difference in crosswinds, between the jersey types. My not-quite-aero jerseys flapped very slightly around the sleeve hems and sides, just enough to remind me I wasn't wearing aero kit.
The aero kit is a bit faster too, especially if I include an aero helmet (rounded, few or no vents, not the full TT/race type with elongated fins or tails). It's usually good for an extra 1 mph over my usual 20-50 mile routes. But I don't go for new personal best times every ride so I don't wear the aero helmet often. It's not as comfortable as my regular vented road helmets. But it also helps with crosswind stability.