Yup, agree that you're taking too many rest days. Better to ride and give yourself an occasional 'easy day' when you feel the need for it. One rest day per week is good, two is OK, but that's it if you really want to build your conditioning. Plenty of guys I train with, notably those who race a lot, rarely take actual rest days.