This is a good question. I have often wondered the same thing but maybe from a different point of view. I too, get little to no physical exertion with my work. On the occasion when i do, i find it a little less appealing to go out and pedal for a couple hours.
I often have wondered how cycling would fit into my life if i had to switch to a more physical job or one which gave me less time to ride. Right now, I have a pretty flexible schedule and can ride every day, which i normally do year round.
That said, something tells me that if i was doing something physical all day, i may be less excited about it, but who knows?