Unless it gets under 50, I'd stick with shorts. On days that are 45-60, I wear shorts and a long-sleeve mock turtleneck shirt from Under Armour (got it at Marshall's for $30, normally $90). Under 45 and I switch to pants and put on some thin gloves. I haven't gone below freezing yet (it's my first year commuting) but I imagine I'll need to add a hat at some point.

Anything over 60 and I'm in shorts and a t-shirt. I might start out a bit chilly, but once I'm going I get nice and warmed up.

And I second getting rid of the backpack. Bungeeing my backpack to my rear rack was the best thing I've done yet. No more sweaty back when I get to work.

edit: just for reference, I'm 5'10", 140 lbs.