Wear a wool undershirt, or bring a fresh undershirt to change into, and put the sweaty undershirt in a pannier or backpack. Even if your clothes don't have a strong odor, others in the office might not appreciate seeing you airing out your dirty laundry in an office environment. It's also possible that some find the odors more offensive than you suspect.