Say hello to your trainer!
For what it's worth, I fractured my wrist last August after getting hit by a trailered boat, and kept riding on the road. It was sore and we have a lot of chip/seal roads down here in the country...but it healed up fine. Maybe a week or so longer than it would have off the bike and it limited my position on the handlebars, but it was fine.