If this is a big concern than I would suggest you have two bikes. For me I really don't care what others think but then again I am old and can be considered a grumpy old man so I get away with a lot. Seriously though if you want to ride a road bike with the road guys then you will need to keep one of them in your stable and consider a allaround for the rest of your time. For me, the top end flat bar/Hybrid bikes are so sophisticated these days that anyone that looks down there nose at them is uninformed. Get what you want and just go have fun and ride.
Frank.