Flip the stem if you want to, you may find that you have a slightly stiffer bar as you have less steer tube to twist on. Who the hell cares if your stem is flipped? Really if someone has the audacity to comment on something so stupid then smoke them, that will end the conversation. Why bother with someone who rides for the fashion statement? You will find in time that only newbs are concerned with such trivial crap.