Depends. . .
I've ridden the entire GAP/C&O 6 times and haven't taken a dip, despite looking, casually, for swimming holes. Others I've seen going off trail to swim seemed to be dealing with rugged,steep bushwacks, and I ride and camp solo and haven't wanted to risk injury. I will say there is a public swimming pool in easy eyesight to the right of the trail if headed from Pittsburgh to DC, in the town of Buena Vista. I have considered offering the staff at the pool a little monetary donation for a shower and a dip in what appears to be a very nice community pool.