I have come to the same conclusion. My legs are nice and toned. Even after the accident. However, my upper body is....... bleh.
Core workouts help one while riding. Plus, with the way I am built, I will never have to worry about excess upper body muscle. Just having biceps and a toned upper body balances the whole thing out.
Looking better in a jersey is also nice.

Our office's new building has a gym of some sort. Being free I sure hope it has some decent weight training equipment but I doubt it. It may be time to join the local gym.