If you are comfortable with endurance geometry on and off road, then it would be a sensible choice. I for one prefer more aggressive geometry and a shorter wheelbase on pavement. Simply swapping wheelsets on my road bike would not be sufficient to make it comfortable for riding on trails. This touches on the point of having a bike that is mediocre at two things, vs. a bike which is built for two distinct purposes.
There is also the consideration of wear and tear on components. I would not want to muck up my Ultegra drivetrain or crack my carbon handlebars while riding trails.