I've noticed this too.
Even before I get up to speed, when I used to commute on a mtb with a pair of cargo shorts and a t-shirt, cars would just buzz by me.
Then I started wearing a jersey and bike shorts and I got slightly more room but they still honk at me when I got in the middle of the lane, not to take the lane but just to go faster and work less in the draft.
But when I started riding my road bike with disc brakes, wearing shorts and a jersey, cars started having a lot more confidence. I didn't even have to go fast, just looking like I'll go fast was good enough for them to let me into that opening in traffic.