I wear board shorts a.k.a swim suits a lot. Many are slightly longer than workout type shorts, all dry quickly. As long as I'm moving, my legs stay warm... it's just the getting started part
If it's even colder, I have one pair of Old Navy pants with legs that have zippers to remove them. They are my favorite, cause even if it gets a little warm during commute I can open up a "knee vent" by unzipping them part way.

I only wear jeans if I'm not gonna be pushing hard and getting hot, but for Jeans and the Old Navy pants I have a strap of velcro (extra from my light mount) that keeps it out of the chain. I don't trust just rolling up since i ride Fixed gear most often.