"The trouble is with experienced and fit riders that they don't always remember what it was like when they were starting out."
Some fit people haven't ever been unfit, so it's not memory, just lack of experience with being unfit in the first place.

Suggestion: Look up all the charity rides within a 100 miles of you and go on them. Based on the ones I've been on, you have a bunch of fast people that promptly lose you, and then you wind up riding with people of similar speed, and there's a bunch of them, too. You're not necessarily riding in a tight group but you're not the only bike on the road either.