I don't know what sort of racers you ride with but I never rose above Cat 5 and my racing improved my riding all hell and gone EVERYWHERE.

Not only could I ride for an hour or more at 25 mph on flat ground and climb hills in the big ring and not hardly feel like I was climbing but I would also go out and do 60-100 mile rides without stopping and only use a water bottle or two and a couple of packs of Gu.

One of the guys I raced with would do 60 mile rides in less than 3 hours on his own and in the fast group they'd do it in 2.5 hours.

I don't think that you've been riding with serious racers. One or two races a year don't count as being a racer.