I would listen to your doctor. Or , if you don't trust him, get another doctor.
But broken in three places may well require surgery. That's way different than a simple single break
in that relatively short bone.