If you are talking your lung, listen to the docs. If your are talking about your ribs, my experience is that you just listen to your body. If the ribs don't like what you are doing, oh boy! They will let you know right now, not two minutes from now, but right now!
This makes sense to me too. I never had activity restrictions for broken ribs, other than tolerating the pain. But a collapsed lung is different. You don't want to mess with your lungs--they're kind of important.