Question is really what is 90% of your riding... Recreational, able to go anywhere when the path dictates ??
I raced and will race again starting next year... I used a Cannondale M900 for commuting to and from work, at the time it was a completely rigid setup... If your rear triangle is rigid and front forks are locked there should not be a big difference...

(1st) what tires are on now ?? That will consume a lot of energy transfer. Unless you are off in the single track, you don't need X-country knobbies, you do need a tire that will allow you traction on dirt but also run at a low friction coefficient on pavement....
Let me put it this way, I trained on my MTB in winter on ice and snow covered dirt trails where you are constantly drifting both wheels, same with this winter as I trained on the road on my road bike with 700 Michelin race tires, in the snow and in the slush...
Tires are important but they are only 50% of the equation. bike handling is also a critical factor.

(2) If you believe MTB's are slow, a friend showed up years ago at a group ride on a MTB, ( everyone else on their road steeds ). end of ride sprint and all the other sprints he won on a flat bar MTB... With smooth street tires but still.... He was also a grad of the former German National Team. ( Yes I have mentioned this elsewhere ) but the point is that focusing on riding is more important then the particular equipment.

(3) Best Case Scenario would be to have Road and Offroad bikes but that may not be possible for everyone...