I've never been passed on a solo ride. However, it's rare that I see more than a couple riders a week.
In my last race, we started 10 minutes before the Women 3's on a 22 mile course with 1500ft of climbing. I was passed by the entire women's field about 10 miles in.
In my defense, I caught and passed some of the women's 4's that started 10 minutes before us. (I was dropped by my 5's group and finished about 15 minutes behind the winner.)
No loathing here, just desire to work harder. There are countless people with more time/money/dedication that can train harder/afford coaches/train with power/etc.
I simply cannot compete with that on 6 (average) hours of training a week and I'm not pretending that I can.