Dude...
Do what you want to do.. don't worry about what other people think... If you're concerned... incorporate a day or two to step off the bike and weight train...

I shaved 30 lbs of muscle and fat off my frame... [I used to weight lift A LOT]
I have a 12.5" bicep from a 14"...
I have a 30" waist from a 34"...
My quads and calves have lost a lot of fat and muscle...

I have completely re-invented myself and my body... I feel great... I feel like my body fits who I am, so it's worked out for me....
My goal is to do whatever it takes to make myself faster and faster...
If I have to have a 10" biceps and 25" quads... so be it... I'll do lunges til I can't walk and burn all available weight off the top end of my frame....
I DO do core-training at least twice a week to stay muscular but just enough for tone... not strength

-Peter