I am a native Californian who grew up in New Jersey. I am married to a Kentuckian. I am white, with ancestors who came to Pennsylvania when it was a colony. The Confederate Battle flag was a mostly forgotten symbol until segregationist started waving it during the infancy of the Civil Rights Movement in the 1950s. That is why it is seen as a racist symbol.
drmweaver2, if you fly the Stars and Bars, I suspect most people think it is a weird texas flag, or something. If you fly the Confederate Battle flag, which is what most people think of as the Confederate Flag, then yes, many people will make assumptions based on the symbol you choose to show. Just as people make assumptions about me when I wear a tshirt with this photo that I picked up on my recent tour in KY, Southern IL, MO and Tn. (I tried to post the picture, but it didn't work. The picture is a picture of President Obama standing in front of a giant Superman statue) Does it bother me when people make assumptions? Nope. I would choose to wear solid color clothing only if it did. BTW, the tshirt version of this photo says "President Obama and I visited Metropolis, IL" It has started several conversations already :-)