If you ride with a brake on your track bike I don't want to get to close to you when I am street racing. You are not committed! Your riding style has an escape hatch.
I want to draft in a pack of wolves that are one track mind.
No problem with brakes, it just seems silly to ride a track bike with a brake. Why not a singlespeed/freewheel?
When I ride my track in heavy street traffic I am more alert, responsive and playing offense instead of defense.
You can work magic with no brakes, but you got to believe.

And if 'your knees hurt' go back to freewheel.