Go to Bike Central downtown and ask them to check it out. This isn't normal. You can throw a chain, but only to do it on downhills suggests something isn't quite right. Frankly, even with a fairly loose chain, you shouldn't be throwing chains a lot.

I'd expect the chain is coming off the cog in the rear initially, not off the chainring in front?

If your wheel isn't aligned with the frame (i.e., pulled to the left in front because it wasn't tightened sufficiently, but not quite touching the frame and rubbing), and your frame lets you have a decent amount of misalignment, you might be having a problem that the rear cog will feed the chain off the top while pedaling and applying forward pressure, but when tension is on the lower chain traverse while you're trying to slow down, tension isn't sufficient to feed the chain properly with tension underneath. Tension on the bottom would tend to keep the chain on the cog and chainring, but above the chainstay any misalignment might show up as a derailment. Too much chain might cause this, but not that likely (on the track we ride chain looser than fixie riders tend to, and chain drops are almost unheard of). I'd guess at an alignment issue.

The only other possibility is that you have a bent link, which might be bent so it doesn't give problems accelerating forward but doesn't like to go onto the rear cog under tension from backpedaling.