We live in southern Arizona and ride year round.
Am 80 years old, weigh 135 lbs and ride 100 to 125 miles a week.
You described your problem and you know what the solution is. Now act on it.
Your best exercise? Push away from the table and get off the barstool!