Keep in mind also, that some of us NEVER get any faster, we just devlop such stamina that we become ultra-distance riders.
A 40mi RTC sounds like a great beginning.
I don't mind getting passed by a fit roadie in full kit, I get embarrased when I'm passed by some guy hauling an entire trailer of camping gear, or two kids.