Looking at your typical ride, I'll comment that doing that, and only that, is not going to build the aerobic base you want for doing longer rides. Especially at your size, the climbing pushes you into higher levels of effort, which burn glycogen rather than fat, and also rely more on fast twitch muscles. You need to be efficient at burning fat in order to go the longer distances while maintaining the performance you want. Your course may make you feel that you are conquering the terrain, but it is not training smart. As an example, I just went through an off-season period where most of my riding was high intensity: training criteriums and fast group rides with fast climbs. The fun stuff. My ability to go 70-100 miles at a high pace dramatically declined, because I trained myself to burn carbs, and became dependent on them. Relying on carbs is very limiting from an endurance standpoint. For me, if my body is in carb mode, I'm going to peter out after about 50 miles, or 2.5-3 hours. In fat burning mode, I do 80-90 mile trining rides and think "okay, what's next?"

So if your goal is to tackle the longer distances, you need to add in some long rides at easier efforts, as that is what will develop your aerobic capacity and endurance. Stretch your course into some of the flatter areas, as opposed to just doing that loop over and over. You are basically doing hill repeats every time out, and that is just not the proper way to train. Include some climbing, but make it a much smaller percentage of your total training. Once you have built a solid base, you'll find you will able to do that hilly route for much longer than you can now, because you will be using your resources more efficiently. It may seem counter-intuitive, but this approach will also help you keep up with the fast guys on group rides. You have to go slow to go fast, or as they say around here: riding slow in winter makes you ride fast in the spring.

Nobody with a competitive streak likes to be passed or dropped - it isn't just a Marine thing. But unless you never ever push yourself, and only ride with really slow people, then guess what? It's going to happen. Getting passed on a group ride isn't the issue. You are all riding together and sharing taking the lead. The issue is being dropped, when you can't keep up. And every cyclist out there that pushes him/herself has been dropped, over and over. Unless you always ride solo, you will be dropped. And if you only ride solo, and don't hide from the rest of the cycling population, you are going to hear "on your left". Embrace it as a part of learning and getting stronger/faster. Doing long easy miles will certainly help you hear it less frequently.

Your posts remind me a bit of a Marine Colonel hired into a company I had helped found. He had it in his mind that, as the new guy, he needed to be the first into the office. Well, my partner the CEO was an early bird, and as he saw our new hire arriving earlier and earlier, he quickly caught on to what he was doing, and came in even earlier, always there to offer a cheery hello when the Marine arrived. Eventually, the Marine realized he'd been had, but learned from it, and settled down to focus on what was really important. We had several Marines in the company, including one of the partners, and General Al Gray was on our board. I've never seen someone's entry into a room have as profound an affect on people as his did: the Marines' backs would stiffen so fast its a wonder they didn't injure themselves.

So take it smooth and easy for a good part of your riding: General Gray isn't watching!