I have 2 fixed gears (Langster, Trek t1) that I only use for "fitness" riding on a trail, and it takes time to build yourself into good shape. With the right gearing, over time you'll notice a improvement in your speed and overall health.

If your really going to ride regularly on a trail, you'll probably want to have shorts/jersey and get a seat bag for your stuff (tube,pump,levers, etc). Don't forget to bring water if its not accessible on your route, I say this because some fixed gear frames don't have mounts for bottles.

Just my .02 if you want to ride on a trail.