How much time do you feel you have to have available to get a good ride in?

I have noticed that since my overall distance has increased I require more time (duh),but with that, I am passing on smaller windows of oppurtunity to ride. I feel guilty for doing it but the shorter rides with more intensity seem to zap me for the rest of the day. Also my recovery seems to take forever, ruining the longer rides I have scheduled later in the week.

Suggestions?