First, carpdeimracing, great read. I could learn much from you.
OP, if you really want to improve, you have to do target training and spread out your work out days. To do back to backs at max level takes some one who is pretty fit and have their body proplery accustomed to that work load.
Do you know about intervals, cadence work, hill climb sessions, tapering, eating right and getting proper sleep?
I say that to show you that riding with a fast bunch a day after you kill yourself alone is a recipe for burn out. Keep it fun and keep growing. The fun is working on your weak spots and seeing inprovements. Spend more time alone working on the disciplines, a day of base miles, a day of intervals, a day working on climbs, speed training day.
You are probably doing too much too soon. Try rding only three to four days a week, but make them quality days. By that I mean have a targeted goal for that day and mark your progress so you can see if what you are doing helps.
Nothing comes easy for me, I'm not a natural, I have to train right, work on weaknesses, eat right and sleep well. I concur with carpdeimracing that, if you do the work, the weight will naturally come off. As far as how much depends entirley upon who/what you are. Are you a serious/determined person who likes to climb? Train in that vein and your body will find it's ideal size due to your training. Are you a hammer who sprints like Cavendish? Train as such and your body will adapt to it, you will weigh whatever you need to weigh to produce the outcome you train for. Are you an all rounder who can do well at everything? Well, we are all jealous and hate you (Ha!).Train that way and you will be that way. But as carpdeimracing says, in every scenario you will end up much lighter than you probably are now and in return more fit to handle what you have trained for.
I'm a short guy, most I ever weighed was while doing body building work (eggs, tuna, chicken, four to five days at the gym), 155lb. At my best while cycling (I'm a climber by inclination), I weighed 129lbs. I lost power on the flats at lower weight than that, but climbed better. So, everybody is different, you could be close to were you need to be weight wise, I don't know, my bigger question is what are you training for and have you fiigured out what your strengths and weaknesses are.
If you just want to hang with fast guys on rollers, train on those same rollers. Since you kill yourself with them Sunday, start Tuesday with a recovery, same route but ride longer than the Sunday ride at 60 or 70% of max. Go hard Thursday, same route but much shorter ride at 80 to 90% and target weak sections.. Saturday you just want to keep a medium heart rate level for thirty minutes to an hour, work on breathing and cadence and DO NOT do anything do build much acid in your legs, nothing hard. You will do that Sunday.
If you are serious about getting better for more than just the hammer ride, train like I said earlier. Find out what you are good/bad at and learn some drills for climbing, speed, cadence, etc.. Cycling has many facets, learn all you can and don't be affraid to try some new things. Keep track of your progress so you know what works and what didn't (like my advice, could work for you, could not, you have to find out what you need, but you find out by doing/trying).
Good luck to you, keep it fun to avoid the burn out, eat better, sleep better. There is so much more to cycling to enjoy than just the hammer rides (I do them too), make the prep work fun and the fittenss will come.