In the cold, you should sweat less.
On a short commute, it isnt worthwhile riding fast only to get hot a sweaty, then take time to cooldown and change.
Ride at a steady pace so you don't sweat.
Dress in layers so you are a bit cool when you start. Don't wear any waterproofs unless it is raining.
You can layer jerseys to form a slight wind barrier or if it is colder, wear a windproof (not waterproof) top.
On my short rides I just use polycotton cargo pants and wear them all day.

For 3 miles, a backpack is OK but dont carry a heavy load. Panniers are better if you have them.