How do you know you're not simply hearing the rough terrain????

There's no WAY I can isolate a click when I'm bombing over rocks.

Assuming you have super-human hearing and can tell the difference between a pebble clicking off your rims or frame, and the sound your fork is making - no, your fork is not supposed to click.