If it's a big enough company, take the issue to HR and say your boss is bothering you by bashing your favorite past time.

Small issue, but in front of HR they'd have to do something about it, I'd hope. As in, tell him to keep it to himself.