Makes sense for you. I keep the same equipment on the bike year round. My weight is flat +/- 1 lb. But you are partially right, I do carry 3 items of extra clothing: fleece, windbreaker, winter gloves. Those are usually left at home during summer, but I still think there is consensus that biggest factors are:
1. head wind
2. increased air resistance due to cold
3. increased energy lost moving the blood thru the body
Humm, maybe I should be happy it's only 1 mph. [ but when you're going slow, there aren't many mph to lose ]