Its called gravity. Gravity applies a force and accelerates you on a downhill, its as if you were still pedalling (but its free). You reach an equilibrium, a steady state free-fall speed, when the force of the wind resistance is exactly the same as the force applied to you due to gravity. Now, if you are drafting behind someone, the force due to gravity which accelerates you is the same for both of you, but since your wind resistance is less than the guy in front of you, since you are drafting him, you go faster.
A group that understands this can really fly down a hill without pedalling at all. You just do a circular paceline exactly as you would on the flats, but without pedalling, and you will fly. The trick is to get off the front as soon as you get up there and stay as close together as possible.
Believe me, you aren't the only one feathering his brakes, everyone is. Try to stay to the outside fringes of the pack on the downhills and use the free speed to get around cyclists and move up in the group.
Il faut de l'audace, encore de l'audace, toujours de l'audace
1980 3Rensho-- 1975 Raleigh Sprite 3spd
1990s Raleigh M20 MTB--2007 Windsor Hour (track)
1988 Ducati 750 F1