I guess if you're a racer you might not want to have a huge, beefy upper body but I don't think keeping some strength and definition would have a huge penalty. Sounds like you like it too so that's another reason not to stop.