56 YO male in good shape, got back into riding a bike a few years ago after not riding since I was a kid. I primarily ride a hybrid and my usual after work jaunt has been ~ 30 miles. This year I bought a road bike off CL and other than a metric century 1 month ago (no problems), I have only rode it ~ 5 or 6 times. The reason is the terrain I ride on is kinda bumpy, I hate the clip in pedals (fell over a couple of times stopping

and also not a fan of the riding posture compared to the hybrid.
Now that the summer heat is here and the local pool is open, I have been foregoing the bike rides more often in favor of swimming 2k meters/day. Normal week would now be 4 days swimming and 2 days riding. I wanted to take out the road bike today, but ran into some issues with my presta pump leaking, so I hunted out some hillier terrain today on the hybrid.
I am going out to Colorado in another month to ride in the Courage Classic, which will be 80 miles/day for 2 days at altitude. My brother has been into cycling for years and has done the ride a few times. I will meet him out there for the ride. I believe the swimming gets me in better overall shape, as I lose weight when I start swimming, However, I'm concerned that I will have problems with the different type of conditioning and also lack of familiarity with riding a road bike (hate the brake locations also).
What do you guys think?
Also looking for some longer casual weekend road rides in central NJ. The ones I am seeing are charity rides with entrance fees (ok with that) but also donations required and I'm already raising for the CO charity ride.