I am unable to train at any other zone but 4 or 5. When I jump on a bike, I have to spin. My cadence is around 70-80 and that take my HR to at least Zone 4. I'm 41 years old so that means around 145-160 bpm. I want to train to do longer distance but I am uncertain how I will achieved that. I would have to reduce my cadence in the low 50's to get low enough and that would reduce my speed to below 20km/h.

don't reduce your cadence, select an easier gear.

actually 70-80 is a touch low, shoot for 90-100, and yes, choose a MUCH easier gear.

FWIW I also have a hard time staying in anything under zone 3, it just feels super slow and like I'm not even working. they say that it will build endurance over time, but I also don't have the time in my schedule to roll around at 12mph for 4 hours, and it's not very fun. I plan on training long hours at low perceived effort this winter on the rollers . . .

How much do you ride? How many consecutive years have you been riding?

This sounds like the normal newbie complaint that resolves with a few months of say 400 miles/month riding.

You aren't spinning. "Spinning" is 90+ on the flat and ~80 climbing until one runs out of gears.

Or maybe you aren't riding a geared bike? Maybe it's a cruiser with flat pedals?

I'm riding a Giant Defy Avanced 2 Carbon. I've been out for 8 month due to a knee injury. I'm trying to slowly come back so I don't injured myself again. I never rode 400 miles/month. I'm trying to step up my training level.

You can substitute a ride with a brisk walk. I found that at least for me, walking (sometimes speed walking) has given me good climbing endurance. I walk between 2.1 and 2.8 miles in a half hour, twice a day on my non cycling days. My dog loves the walks and they do help me with conditioning.
For a few months, I stopped the walking, thinking I didn't need it and just rode more. Not only did my endurance suffer, my dog was unhappy..... almost depressed. It's all good now, try it.

if you want to ride farther and longer, go out and ride longer and farther. it's pretty simple. start by setting a goal (I'm gonna ride 20 miles without stopping) and gradually increase that, without worrying about how long it takes you. then switch it up, and start putting some constraints like 30 miles in 2 hours, or whatever.

100 miles/week is a great goal for anyone, and you can easily get into racing shape with a structured plan with this amount of time, but it will likely take you a few season to get there.

Well, that's it then. Once you get into some kind of aerobic shape, your HR will come down to the usual reported levels. One must understand that training suggestions are for those who have been training. They aren't applicable to everyone.

//snip

Quoting your age in a reference to Zone 4 heart rate suggests to me that you don't know what zone you are actually in.
When was the last time you measured your MAX bpm?

There is an established relationship between heart rate and age, but it is imprecise. Toss the formulas out the window. My own MAX is 192 (for this year) and I'm 57 y.o.
Your MAX could be much higher than you think. Check it.

I agree with the rest who suggested higher rpm @ lower gear. Aim to maintain at least 90, and that includes while ascending.

I went by the calculation provided by Garmin as to what is my Max HR (181)

If you want to really set your zones accurately, max HR is not the way to go. For one thing, even with the best intentions it's almost impossible to determine. Most cyclists set their zones off their lactate threshold, which still isn't easy to determine, but at least it's fairly accurate.

Here's a link for a good system for finding that and setting zones by HR as well as by power: https://trainright.com/cts-field-test...-calculations/

If you haven't performed your LTHR test to determine your lactate threshold, you're just guessing as to what your zones are. You're only in Z5 if your LTHR is like 140, which is highly unlikely. Everyone is different, but for me, 145bpm is smack dab in the middle of Z2. My Z5 starts at 171bpm, and I don't even get into that zone most days.

Also, higher cadences bring down your HR. You will likely have a lower HR at 90rpm than you do at 70rpm. Mashing puts more load on everything, and your HR rises.

Even though zone2 is zone2, everyone's personal idea of each zone is a bit different.
For me zone 2 is just about a fun recovery ride when my legs and system aren't feeling totally beat up. That would be 105 to 124ish HR. my AT/LTHR is 160-162ish (old pharte...). If I'm real knackered I'll take it down to 95-110 and only do 45-50 min.
Max HR is really hard if not impossible to really peg down... I prefer to use AT/LTHR. As noted by others do some accepted LTHR testing, then if you must have 'zones' you can set the division point between ZOne 4 & 5 at the LTHR. Zone 5 is mostly some considerable level of anaerobic. Its not an on/off thing. At 165-168 I can ride for quite some time, but eventually I will lag - at 175ish I will detonate fairly quickly these days.

To get down to zone 2, for someone who comfortably rides 90 to 105 rpm, drop to 85-90, flat profile ride, pick whatever gear gets you there at that cadence (or you can spin higher if thatz is your thing). Whatever speed, zone 2 is not about speed. Rises, light uphills will take you a bit higher for that brief period.

As a fly-weight climber type, in good shape you can manage 17-18 MPH on flat ground at an endurance pace.

Some bigger guys have a 20 MPH all-day pace.

Most of getting good at going slow comes from going slow, with power at lactate threshold the cherry on top.

The important thing is staying below your aerobic threshold so you stress your oxidative energy system and slow twitch fibers, not the time spent doing it.

Going harder brings your glycolytic energy system on-line with the shift sticky.

I'm 43 with an aerobic threshold around 140 bpm and lactate threshold heart rate of 168 which would make Friel Z4 158-168 BPM.

Ride as hard as you can for half an hour at a steady pace. At the end you should have nothing left. Try a few times until you get that right. Your lactate threshold heart rate is the average over the last 20 minutes (it takes a while to catch up to your output)

Friel starts Z2 at 81% of that, Z3 at 90%, Z4 at 94%, and Z5a at 100%; although you'd be better off using your aerobic threshold AeT (where conversation doesn't flow, breathing becomes rhythmic, and lactate starts to accumulate. It's about what you can manage for a 5 hour steady-state effort with an even split) as the limit for endurance training.

Perhaps not coincidentally, AeT heart rates are close to what Phil Maffetone's formula spits out.

I had that problem, which really isn't a problem so much as a preference that you might wish to control. I take my 29'r on about half my rides, max gearing is 32x12 and controls the speed for the flats, on climbs I'll naturally gravitate to whatever I need for comfort with gears to spare (unless you can spin at 150+ rpm), so it kind of kills the urge. For lower cadence I go for a fixed gear at 51x13 (short rides), the rpm's stay fairly low overall, but heart rate stays pegged (almost impossible to climb slowly with those gears) and the speed seldom drops below 20 mph.

50rpm to go 20km/hr? That's a 50/53 big ring gear.

Assuming you have a normal geared bike with a 34/39 inner ring, which it sounds like you do, you should be able to go at the absolute slowest 18-20 km/hr on 100rpm if you're in your easiest gear. If you have climbing gears you might be able to pedal 15 km/hr at 100rpm. 7-8 km/hr at 50rpm

145 to 160 would be about right for zone 2, for a 41 year old. Go to your local college or athletic center and have your Threshold power and max HR tested. It isn't that expensive. The internet formulas are not accurate. And people rarely go hard enough to determine it themselves. You need to suddenly feel like you are going to vomit while at the same time near passing out. It often takes a competitive situation or lab to properly determine max HR.