I'm not an MD, but bone will generally start to knit around 3 weeks after an injury (several cracked ribs taught me that). Note that that it is *not* strong after that time, merely fairly rigid.
I have known people who cut off their casts to race motocross. I've also know people who did that and now have permanent issues with the legs.
It's not safe to be on the bike with a cast, and you're risking further injury. Don't do it.
Or, you may apply my alternative metric, which I like to call, "How stupid would I feel if <x> happened?"