I was self-conscious about people thinking that at first when I was car-free. Then when I first started at my current workplace I used to make it a point to drive in at least once a week, just to dispel that notion.

However, I think that most people don't make that assumption and that your coworkers are anomalies. When anyone brought the subject up (they don't anymore), they'd tentatively offer something about saving the environment, staying healthy, saving money etc and I'd just laugh and say naw, I just like to ride the bike. It might have been in the back of their mind, might still be with some, but I seriously doubt that anyone's jumping to the worst conclusions. If they are, then if they didn't have that to home in on it would just be something else.