I would think it would matter how fast you are going. Roadies generally are really moving, people who are Freds just are poking along, a little faster than a walk or run. I haven't noticed a lot of joggers with helmets - but if you are ripping down a hill at 40 MPH without a helmet - you might not have anything to worry about since it just might be that your brain isn't working that well anyway.

A bike to me is on the edge - at lower speeds, not much to worry about - but at high speed - you can crack your noggin good.

I personally don't wear a helmet very often (almost no one does here) - and I keep my speed to no more than 20 MPH. I suspect overall, that is safer than having a helmet and going at speeds that are much higher than I am comfortable with.