I have not noticed that problem. In most cases I would say that folks I met on tour were open and friendly toward me. It is obvious that I am in bike clothes, but I would not have thought that would make a lot of difference. FWIW: I wear bike shorts and bike jersey in a fairly conservative color. I am often sweaty, dirty, and unshaven, but that doesn't seem to put most people off.
I think that the biggest thing is how you act toward others. Being open and friendly goes a long way if it is genuine. If you are working at being friendly it can come off wrong and put people off. Just be your best self if that makes sense.
Where you tour makes a difference too. Most of my experiences are in rural areas and small towns. The big cities I have toured through were not east coast cities. I suspect that may also make some difference.