There is a fair bit of play in my variable angle stem at the pivot point.

I have rode the bike for a bit without any issue such as the angle suddenly changing. However, there is definitely give where the pivot is. I can wiggle the handlebars up and down a bit. But I haven't been able to wiggle the handlebars past a narrow range; I can't permanently change the angle just by pulling or pushing on the handlebars as I ride.

Is this normal? I'm guessing it's just loose tolerances?

I would try to tighten the allen bolt further at the pivot point. But the allen bolt there is gigantic and I don't have anything quite that big. I'm planning on heading to the local co-op anyway to borrow a tool this week.

In the meantime however is this safe/normal?

No, that is not normal. Needs tightening. Safe? Depends. Don't be jumping any curbs.
You should be able to find a cheap set of hex wrenches that would do the job at Harbor Freight, Walmart, etc. etc.

There are a number of brands and designs. I've seen some that have some play or movement no matter how tight, and others that tighten tight. If you have some mechanical skill and understanding, consider taking it apart and seeing if you can fix the problem, or if the design is positive for movement beyond the trace play (toothed design or similar).

I might try a beer can shim. FB feel free to jump in with guidance on this one.
Cheers,
David in Alaska

The ones I've encountered with play in them usually have a clamping plate with a bolt under the front of the pivot like the one in this picture. It can be hard to see if the stem is pointed down. Tightening that bolt usually gets rid of the play in the stem.

Most every type of hinge clamp I've seen could develop play. Sometimes this play is reducible with tightening. There is a reason that these stems are not intended for off road use. In time this play could become a safety problem. The more loading they see the more likely the problem.

Best fix is to get a one piece stem that duplicates the bar position and get rid of that flexible one. Andy.

Is it play or just flexy? I had one on a hybrid that never felt solid. I just used it to find the angle and length I liked and replaced it with a cheap solid stem. Best upgrade I did on that bike.

This+ In my experience, if the bolt won't remove the play.. the stem should be tossed. No reason to risk it over such a cheap part.

If it's anything like this one Lickbike.com | Dimension 26.0mm adjustable road stem - 110mm) :
Unscrew the bolt a little, enough to make it looser but not enough to adjust the angle. Put some weight on the bars just like you were riding, push in the cap on the far end of the bolt head to make sure it's not loose, then tighten the bolt as much as you can with your hand.
At least for mine that took out the ability to pull on the bars to make it click.