You should be able to do back to back days of threshold work which is what you would be getting on long hills if you were going hard. I suspect your problem is nutrition related. If you're doing a 3-4 hr ride with some intensity you will definitely need to pay attention to replenishing your glycogen stores if you want to workout again the next day.
Start by eating/drinking about 250 Cals/hr of carbs on your rides. For guidance, you're probably burning somewhere between 600-800 Cals/hr on the bike. Some of your energy will come from fat stores, but without knowing your intensity, it's very difficult to determine how many carbs vs fats you are burning. If you want to ride hard the next day, just assume all calories burned were carbs.
Assuming you start with roughly 2000 Cals of glycogen available a hard 4 hr ride could easily exhaust all your glycogen so you would need to eat at least 2000 Cals of carbs after the ride. Pay attention to eating something immediately after the ride as your body will assimilate carbs faster during this period.
You will need to experiment a little to figure out what you need. Initially, I would try more carbs than you think you need and see if that helps.