Ride more frequently than just two or three times a week. Right now you're jerking your body back and forth between building-up and knocking-down. Your between times are long enough your body thinks the need for building-up is past. Then you smack it again.

The whole reason your body is rebelling on the second day is that you haven't trained to ride more frequently.

All the other advice is good too. Follow that after every ride, but increase the number of rides. At the very least, reverse the proportion of ride days to rest days. Ideally move to five or six ride days.