Yes, you are crazy to use fixed offroad - esp brakeless - if you mean offroad as any trail that isn't paved/MUP style.
Just to second some things already mentioned above, if you are really riding a MTB trail, go freewheel (pedal strike on tree bits/etc to clear is the thing I would fear) and use brakes.