Far from it. I wish I had more money so I could afford to travel to all the events I want to do.
I focus mainly on one-day events, so all it really takes is one day a week to focus on a long ride. The rest of the week can be spent doing 1-2 hour rides as a mixture of intervals and recovery. Like Carbonfiberboy said, commuting is a good way to get that time in. Otherwise, how much time a day does the average guy with a job and a family spend watching TV or things like that? It just takes some commitment, time management, and an understanding family.