thing to worry about is how bloody dirty the pants are gonna get when you're on the bike - you'll probably wanna get off the bike and shower and then put your "casual" pants on right?
fwiw i've got a pair of uber lightweight, fold-down-to-nothing khaki sort of things made by an aussie brand called gondwana what have zip-off legs so they double as shorts. cost me about AUD 50. good for off the bike. on the bike i've got two pairs of shy shorts - ride in one pair and then, if shorts are needed for the night, chuck the other pair on then ride in them the next day. rinse, wash, repeat.