If it's single speed (not fixed) you're going to want to leave the derailleur on there, but restricted to where the cog is, so as to take up chain slack. That is, unless you have horizontal dropouts, but it doesn't sound like you do. You can also get a cheap single-speed chain tensioner that's slightly more elegant and lightweight, but that's another.. $20 or something. Do not use this if you're going to go fixed, it can be done, but it's inelegant and probably dangerous.