I think the best advice I got was to REST. So, I started out basically riding 3 days a week too, though I was riding a bit further than you (typically 12 miles, rest for a couple hours, ride back).

I would usually do a ride where I really pushed myself for speed and put the 2-day rest after that one. I did the same later when I sometimes tried to ride further than normal.

I now ride more like 5-6 days a week and more often take longer rides, but the rest time still matters.

Another bit of advice I got was to try to do a longer ride once a month and add about 10% to that distance each time. This makes the normal distances seem easier.

Bob