Skip the spandex if possible or use MTB shorts. Throw the hi-viz stuff in a pannier/backpack before you walk in. Otherwise do what you're doing and don't worry about them.

I'm lucky. Commuting by bike is pretty common in this building and some people wear full kit so my solid color spandex shorts look only moderately "gay" by comparison.