That cast is gonna stink something serious as it is. Going riding with sweat running down your arm into the padding inside the cast won't help the situation.
Putting that kind of pressure on the wrist you're trying to heal sounds like a bad idea IMO. That said, I did ride MX with a broken wrist in a cast when I was younger and slightly less concerned with the possible negative consequences.