Normally riding with the fast group and just trying to hang longer will make you faster. However, if you're only hanging for a few minutes you're not getting much intense work.

I'd drop 2 of the 3 group rides, and replace those with intervals. First, I'd work on raising threshold, so intervals, 10 to 20 minutes long at FTP. After a few weeks of those, then I'd work on speed with shorter intervals, such as one minute on, one minute off, 2 sets of 5.