I wouldn't feel bad, I'd ride more. I work 2-3 days a week as a temp for a express purpose of having a more recreational life. I don't make as much money as I could if I had a full-time job, but I'm happier this way.

I used to kill myself working until I was 25 and almost died from tonsilitis that I ignored in order not to miss work. I almost went deaf and still have muscle damage in my face from the spread of the infection. It taught me a valuable lesson. Enjoy life, work just enough to pay rent, eat and save a little for a nicer bike.