My suggestion is that you look at a second wheelset for offroad use. Offroad specific tire, including cyclocross, are made of a softer compound, and have deeper tread patterns than road specific tires. That makes them wear faster and increases rolling resistance on the road. It also makes them very effective off-road. 'All rounder' tires are a compromise that do a bit of both and nothing terribly well - thats what you have now.