First , make sure that your brakes are well adjusted and do not stick or catch on the rim.
I only use the rear brake for emergency stops or to reduce my speed on long descents (where I alternate front and rear).
You can brake with the front on downhills but you need to be very responsive. If you feel the wheel starting to lose traction you should release some of the braking pressure.