We take the front rack off except when we need it for touring. We find it gets in the way and slows us down.
We usually leave the rear rack on - we do longer rides and appreciate having the rack with a trunk bag to hold rain jackets, snacks etc. We have several size trunk bags including one that has zip down mini panniers for even more storage.