Increasing avg speed is a poor objective. It leads to riding as fast as you can all the time, which is ineffective training for racing unless your goal is to tow the field around until they go hard and drop you.

When training for racing, you need to go really hard when its time to go hard, and really easy when it's time to go easy. "base" training means Z2, not as hard as you can go.

If you've only done 1000 miles since March then suddenly doing multiple 3-4 hour rides is going to make you tired. You will need to work up to that.

I would not suggest starting base now. You want to periodize your training to fit with the racing season and your A races. (See Friel's training bible book). Instead, try group rides now. Do some interval work (2x20s or something shorter) one day a week. Do a long (3-4 hour) endurance ride. An easy ride between your weekday block and the weekend when the group ride is. JRA for the rest. Don't ramp up the volume too fast.