Another option, triathlon shorts. They've got a chamois, but no padding, and are meant to dry quickly.