Simple uncomplicated collarbone fracture. Expect a simple uncomplicated recovery.
Could it get more complex? Of course it could. Every commenter who has a story is the evidence for that. But 99% chance you are clear. Your doctor is not excited. Three weeks is longer than usual for the follow up. He expects you to cancel that appointment and go for a bike ride.
After the first time most don’t bother with the doctor for subsequent collar breaks. Also, if you have a simple uncomplicated break you fell correctly.