I have been training fixed for a very long time and there are a few things to help you along.

1) Racing on the track has given the ability to have a very high cadence. This is terribly helpful going downhill as you do not want to bounce about the saddle at speed. Work towards a high cadence while having a good position in the saddle.

2) You most definitely have to run a front brake and if you deem necessary, run a front and rear. I ride all my centuries fixed on both flat and rolling courses. You have to save the kickers for going up the buggers. Do not waste your energy resisting the pedals to slow on the way down. Scrub the front brake or both as needed and let your legs "relax" while getting to the bottom of the bowl.

3) Run as big of a gear as you can comfortably. When I first started training fixed on the road back in 2004 I ran the 48x16 that came on my Bianchi Pista. I got stronger pretty quick and now run a 54x16 on all my training rides and flip the carbon in the back to the 14t on the other side on calm days or on flatter courses when I'm in somewhat shape. The bigger gear really helps the cadence going down the rollers.

In any event, good on ya for training fixed. You will most definitely get stronger and faster. It's inevitable.