I have lifted far longer than I have ridden. I think the two complement each other. But, I am with you—I only train upper body with weights.
I think the positives out weigh the negatives. For example, as people age, they experience bone loss. Weight training increases bone density.
The only negative I have experienced is sometimes the day after a heavy tricep workout I feel the stretch in the upper arms while keeping my hands on the hoods.