You don't say how you determine what your zone 4 is but after a 1 hour effort with 85-95% of the ride @ Z4 you should be beat up. I have done testing using power to determine my training ones. My zone 4 efforts could never last more than 1 hr and then it would be the hardest workout or race done in a very long time.

To get a harder workout completed you could hammer the hills during your 1 hr rides at a high intensity, like the hardest you can go for 2 to 4 minutes then backing off the pace to recover while riding, those type of intervals would boost your VO2 Max.