dont remove the brakes. My first fixie had no brakes and I have almost been hit a good amount of times because of it. My bike now has a front brake and riding is much more enjoyable. I'm acctually gonna get a rear brake soon here, riding down some of the mountain roads here is still pretty nuts just having a front brake.
But yes the Madison is a nice ride. I was looking at getting one a while back.