OP, you'll be getting confused by now.
If I had to have only one bike, it would be a light tourer or a CX bike. I couldn't be competitive on them in a crit, or riding very challenging terrain off-road, but they are versatile enough to do pretty much anything else, and enough fun to be an enjoyable ride unloaded. You'll definitely notice the difference between them and your mountain bike. You'll be a little bit faster, less because of the weight than because you can get more aero in the drops. And, in my opinion, you'll find that they are more pleasant to ride over longish distances.
Which you choose is a matter of personal preference. I've been used to riding tourers for a long time, I like the relaxed set-up. Something not too heavy, like a Raleigh Clubman, might be suitable but there are plenty to investigate.