Personally, that would drive me nuts. It does sound like it's a case of "gunked up sticky pawls" in your freehub. Yes, you can ride it, but as with any mechanical mechanism, if it's not working properly then that is a detriment to it's functionality and longevity. And, because of that, it also MIGHT have an impact on your safety at some point. If your freehub decides to completely lock up, let's say, while your coasting down a nice long, fast descent.......your bike instantly becomes a fixed gear. That'll be awfully fun in a corner, don't ya think? The chances of that happening? Probably pretty slim...but I'd take no chance for a slim chance any day.