I ride 3 or 4 times a week. Rest days are just as important. Sunday is the long ride of 40 miles + Then Tuesday do a 20 miler on a flat course just a bit faster than my Sunday time. Then Thursday it is hills with 2,400 to 3,000ft of climbing on a 20 mile route. May go out Saturday on a flattish route with just a few slopes in it for 20 miles to the LBS.

When I am fit enough- I will go for a few extra miles on the Sunday ride- hoping to get to about 60 miles and on the hilly ride I will sprint the last 200yards of the slopes or hills and then recover before I attempt another sprint.

You don't say what mileage you are doing but getting out for at least 1 hour and keeping the legs and lungs working firmly does improve fitness. Don't mean sprinting for all that time but getting so the legs are "Just" feeling the strain and the lungs are working hard is the only way to improve. You may not feel that you can do that at your current fitness but there is nothing to stop you pushing hard for 5 minutes at a time---And then recovering fully.