Many newbies think that everyone is staring at them when they ride. They take random facial expressions on other cyclists, pedestrians, and drivers and interpret them into something those expressions were not.
These forums are full of posts of the newbie riding along normally and all of a sudden a driver is scowling at them and mouthing something. Or another cyclist approaching from the opposite way gives them a dirty look. Or a pedestrian has a look of horror on his/her face. There are lots of posts where newbies mention being stared at as well.
In most of these cases, there's another more plausible explanation. The driver could be in the midst of singing along with the radio, the cyclist could be squinting into the sun or dealing with a bug in the eye. The pedestrian could be stifling a yawn. And many people gaze blankly into space (looking like they are staring) as they try to remember grocery lists or try to figure out a problem at work.
Most of the time people are too preoccupied with their own lives to notice things happening around them ... especially if it is something like a person on a bicycle quickly cycling past. People walk right past friends because they are lost in their own worlds.
And have you ... as a pedestrian, driver, or other cyclist ... ever tried to see what kind of bicycle a person is riding? It's not that easy when the person on the bicycle is actually riding. When you are on your bicycle and riding, it's not that easy for others to notice any details about you.
When you get off your bicycle and walk somewhere, some people do feel self conscious and in some cases it is inappropriate to be in cycling attire. It's easy enough to carry a pair of basketball shorts or something similar if you know you're going to be in public places off your bicycle. When we toured France in 2007, I felt more comfortable there slipping a pair of basketball shorts over my cycling shorts when I was walking around. And I wore a nice-looking wicking T-shirt rather than a jersey.
But ... to the OP (who has probably long since dealt with his issue and moved on) ... if you're just nipping in to pick something up somewhere one in a blue moon, just wear your cycling gear.