It's a combination of the tires and your conditioning. Swap out the tires for some wide, semi-slick ones and you should see some improvement on road and notice very little (if any) negative performance off road. These are the ones that came stock on my bike and I have no issues doing the same type of riding that you describe-
http://www.amazon.com/Tioga-City-Sli...a+city+slicker.
You should have little difficulty in swapping out the tires yourself. If you don't know how, search out how to vids because removing the tire from the wheel is a key element in fixing a flat.