I broke mine this summer, so here is my experience. If the fracture isn't a nasty one, you shouldn't worry. You will be able to sit on a trainer in 10 days, and on a real bike in 20. Full effort level after 30 days.

Unless you are an elite cyclist, you should not notice a big drop in fitness level.