How long/hard a ride are we talking? How much of a "sweater" are you?
Can't you simply wear your normal clothes and take it easy on the ride in so that you don't get sweaty?
I ride to work in my normal clothes. Mountain bike shoes that take clips but are fine to walk around in and look pretty normal (Shimano MT22). Rain pants, booties, rain jacket over it all.