I wore out a couple pairs of cutoff blue jean shorts riding bikes, so it can be done.
I'll second or third the idea of bike shorts. It only takes a few minutes to change when you get where you're going. If you're too shy for the spandex crowd, get some mountain bike shorts instead.