I drove 2-1/2 hours yesterday for a 50-mile charity ride, for the 2nd year in a row. I was off the bike for most of last year and really wasn't all that prepared for 50 hilly miles but I couldn't justify the long drive for one of their shorter (10,20 mile) routes. Like someone else in this thread said, the ride has to at least equal the drive time to make it worthwhile to me.
That said, most of my rides are out my front door. Occasionally I will put my bike in the back of my truck in the morning to ride the rail trail after work, but I find trail riding is only fun if you're riding with someone - if going solo it's quite boring.