To the OP, I do the same thing and I guess it really doesn't matter. I ride alone exclusively (except when riding with my 8 year old daughter where we do about 8-10 miles maximum, at about 8mph average). The last time I rode with a friend was in 2006 when I first started "cycling" and doing a 15 mile ride was HUGE for me. Looking back, that friend who was really into cycling was very diplomatic and understanding; I now know he kept a MUCH slower pace for my sake.
Nowadays, my standard ride is around 30 miles at 15-16 mph average and I'd like to think that's pretty decent for doing mostly road on a gravel bike with 40mm kinda knobby tires. I'm 6'-3" and currently weigh around 240 lbs. I always gain weight over winter and lose it in the summer when riding a lot; by fall I'll be like 210 and my speed will be like 16-18 mph.
The other day I had a nice tailwind and was riding at a nice steady clip of 21-22 mph and I saw a girl gaining on me. It took a few miles but when she passed me, it seemed pretty fast (I guess 25-27 mph). I thought I was doing really well and felt a little disconcerted but I got over it quickly. She was on what looked like a pretty nice Trek Domane, and I probably outweighed her by 120 lbs, and was probably close to 20 years older than her. In other words, I think I do pretty well for an older, big guy so I try to keep that in mind when comparing to others. But the comparisons can surely hurt you. I see it in my profession all the time and have to really stress it with trainees or it becomes a self fulfilling prophecy and can lead to failure.
TL;DR: Just ride how you ride to enjoy the sport. Min/Max numbers games take the fun out of it. Just do your thing and have fun.