The only real reason to change your entire bike is to change geometries. Most other things can be handled through retrofitting.

My 2c is to try getting "real" slicks (26 x 1.5 or so, and not just the cheap one on the rack), good brakes, and a bike shop tuneup to keep your bike running super-smoothly. Then see if you want to play with different handlebars or bar ends. $400 goes way farther in customizing a decent bike than in buying a new one. You can even consider changing out your drivetrain. Indeed you could buy an internally geared rear wheel (be sure to match your dropout width) and a new front chainring, and find that it improves your experience dramatically.

If you wanted to upgrade for real, $800 is a decent price point, but for some reason, $1k seems to be where Really Good Bikes start to happen. Something like the Surly Long Haul Trucker (LHT), for example. Of course, a used LHT sells for less . . .

My contribute on the road vs. mtb discussion is that I find that the more aggressive riding posture is unnerving in traffic for me. I like to be able to see from a little higher up. But that's my preference, yada yada.