I think I know what team you are on, and it's a large one at that. I am sure there are others on your team that either get dropped or don't go to team rides for fear of getting dropped. Maybe you could do your own unofficial team "B" rides?
I hope your team mates don't leave you when you get dropped. When I did rides with my previous team, those who did the dropping would slow down at the top of the hills so those who got dropped can catch back up. It's bad form if they leave you in the dust. Come to think of it, your team leader isn't exactly a hill speciallist (a cat 4 crit speciallist, yes)
As for that cat 4/5 road race (Fallen Leaf), that was probably not fair to you and other new cat 5's as it was won by your cat 4 sandbagger crit speciallist and had several people on the cusp of being cat 3's who should have done the 3/4 race instead. Combine all that with the shortening of the race (I think our race was only 35 miles or something) the speed naturally will go up.
For what it's worth, I wear my team kit on every ride.