I could care less what others think. I don't bring it up unless someone asks me about it. They all know my passion. They see how much time and energy I put into it. Some people are envious that they have no passion about anything - unless it's sitting on the couch.
But try to surround yourself with like-minded individuals and you'll feel better talking about it