It's up to you. Different people have different tolerances. I have talked to people who are absolutely terrified to even think about riding on the roads I ride on - while I think they're extremely safe and I've never been even slightly worried about it, except in the winter when the road hasn't been plowed, and cars think it's still OK to do 60 MPH when there's two tire ruts down the road and I'm in one of them.