You'll be fine.

What I've found funny is that if I go for a long ride the day after an upper body workout (especially a chest, shoulders, and triceps day), it can get harder to hold up my body than to pedal.