I think it depends on how long you ride for. I'm like you in that I only do upper body workouts and figure the cycling is keeping my lower half in shape. I've noticed that after doing bench press and arm work I seem to get more fatigued in my upper body if I go out riding the next day. Only seems to happen after I hit the 40ish mile mark. Then I start having a harder time keeping my hands on the hoods and maintaining my position on the bike. Because of that I try to go a little lighter on my workouts if I know I'm going for a long ride the next day just so I don't run into that issue.