Now that I see your solo ride, I'd say having both types of rides each week helps you rather than hurts. Most of the training plans I've seen usually have a nice mix each week....some days of Endurance miles, some days with VO2max/Anaerobic interval work, etc.
If you're worried about how you're spending your time, I'd pick up the Time Crunched Cyclist book and pick one of the plans in there. For whatever its worth, every Sunday it suggests a non-super-hardcore group ride (or similar solo ride).
PS, the TCC book is what I've used to work my solo average up to 20mph (if I'm really going for it). I seem set a lot of new personal bests when I'm 6-7 weeks into the 12 week program (I'm finishing up my 2nd time using it over the past year).