I almost always ride from home even when meeting up with a group or doing a local organized ride. The furthest I've ridden to a ride start was around 35 miles.

If riding is important enough to you, then if I were you, I wouldn't buy a house that hindered my enjoyment of it. I don't like to drive my bicycle anywhere, which is partly why I rarely ride my trail bike, so it would be a no go for me. If you love the house, that is a consideration though.