That rules out one possible reason you are feeling unsafe while fixed. Get a freewheel and a rear brake and hopefully you'll be happier. People seem to be recommending White Industries Eno freewheels.

Fixed gears are more dangerous when it comes to servicing them because you can get finger caught in the gears etc. They are also catastrophic if you get your pants leg stuck in the gear or something while riding. People can have trouble getting into and out of foot retention because the pedals are moving which can make them dangerous and some people find that spinning at high speeds going down hill is at least disconcerting, so you are not the only person to decide that single-speed is safer.

edit: Also cornering fixed is somewhat hairier as you can get pedal strike.