I want to convert a steel mountain bike to a commuter. I will be using moustash bars. should I use one size larger frame? I ride a 17in now, should I got to a 18 or 19in?

That depends. If you want to ride in the forward curved part of the bars a lot, I'd not go a size up, as you'll be pretty well stretched out. That kinda depends on the era of MTB you get as well. Many mid-late 80s frames had shorter top tubes than many mid 90s frames, so you might want to go a size up if you're gonna convert a 1986 bike. How does your 17" fit you? Try resting your wrists on the bars and pretending your hands are gripping the curved section of a mustache bar. Does that seem like a useable position to you?

How aggressive is your position on the 17"? If it's a pretty flat-backed XC racing position, I'd probably stick with the same size frame, perhaps get a shorter stem for the mustache.