Seriously? Turns out I've had asthma pretty much my entire life though I didn't know it until about 20 years ago.
On top of that, it also turns out I'm working with less than 70% of normal lung capacity for my age.
If I know a ride will include some serious effort, I take two preemptive puffs on the inhaler before setting out. It does what it can.
Long, hard climbs can stop me cold so I take a lot of -literally- breathers until I'm not gasping anymore... then off I go.
Once I realized it's "built-in" I stopped thinking that my fitness was the problem. Stay within your means but don't let it stop you. Ever.