You can do it, but the chain, chainrings and cassette may be ramped and pinned (designed to shift gears easily) which means you might drop the chain on occasion. All that crap weighs too and looks crappy but thats not really a functional concern.
You can do it to a bike with vertical dropouts if you use a chain tensioner pulley, or if you find a combination of gears that ends up being the right total chain diameter to have a taut chain without a tensioner. Half links can help alot with this.
If you have ramped chainrings it may be a good idea to leave it on, as there is chance of dropping the chain. If you aren't using a tensioner it becomes less necessary
The rear hub is probably cheap and gunked up as hell. The grease has probably hardened keeping the pawls of the freewheel engaged. Some cheap or brand new hubs do this, but its not really a concern while riding. Either that or you have a magic multi speed fixed gear hub.
If the bike has bolted on chainrings you can remove the chainring you aren't using and then shorten the chainring nuts or just buy shorter ones. If the rear hub is threaded you can guy a BMX freewheel and thread that on. If the rear hub is splined, you can buy a spacer kit and put whatever cog you want on it. This all costs money but will be better in the long run.
But in conclusion, if all you are going to do is take off the shifters and derailleurs, theres probably not too much reason to do it in the first place... Just keep it in one gear and be glad the extra gears are there when you're hauling groceries up a mountain.