//Rant warning!
I think what ticks the OP off is that average (small minded) person thinks that anyone who rides a bike to work must have either lowly economical reasons or some higher, profound environmental or political reasons to do so. It doesn't occur to them that for most people biking isn't a tool for making political or social statements (although it can be) but that most people do it just for fun, it's just beyond their small minds' grasp. People associate non-recreational biking with weirdness, fringe activity for strange people, they don't get the basic ideas behind utility and commuter biking: fun, simplicity, freedom, etc. They also can't grasp the fact that some of us want to live away from all the "goodness" that our wonderful civilization is throwing at us - except for the latest in bike technology and outdoor gadgetry

It's beyond them that someone would prefer to wander around woods to see the real world rather than to sit in front of 60" TV, or that someone would rather pedal with 50lbs of camping gear on their racks, rather than driving a "comfortable" SUV.
Although, I also agree that the OP gets a little too bent out of shape over this. Relax, people will never get it. For me being green is like tertiary reason, after all others, a side effect. I don't believe that I'm making any difference but, to some degree, I enjoy the thought of not adding any more to the damage being done to nature. However, recently I started making more effort to limit the use of my car and I think not burning fossil fuels for short daily errands and saving energy in general is becoming a source of quiet satisfaction and maybe even pride for me. Although, officially this is still about fun and fitness. In the end cycling is a combination of many things, some deliberate and some incidental.
So, next time someone says that to you, think "yeah, I do it for myself mainly, but incidentally yeah, I am helping the planet a tiny, tiny, tiny bit, and you don't!" and smile

Let the small minded people walk away with their happy thoughts, no need to change that, and don't let them spoil your fun.
I know it's easy to say, but myself I've been on a quest to become impervious to stupidity, arrogance and small-mindedness while cycling and beyond. I'm working on ignoring this kind of people, treating them as obstacles and noise. I was getting too worked up about arrogant and disrespectful cyclists and pedestrians, but I'm learning to rise above that. Let them live inside their miserable bubbles, unaware of the world around them. It's all background noise. Enjoy the ride, and know that what you're doing is right and good for you and don't waste your energy on trying to educate and enlighten those who are not capable of being educated and enlightened.
//end of rant