I really suspect this is your imagination: people are cool with you riding a clown bike, but suddenly get all worried if you are in lycra? I suspect the difference is you, not them.
Where are you? In London, I don't think you'd get a reaction to a folding bike of any sort, while in Melbourne (for instance) the small wheels and the fold draw the eye no matter what.
Personally, 7 miles is around or just below the threshold of wanting to change when I get to work, but you may pedal faster.
(bhyte, I agree with your point, but it;s not the most egregious offence of that type even this week!
http://www.bbc.co.uk/news/uk-24278768)