I've got less than 2mm clearance between my rear tire and the brake caliper. Front is more critical, because you don't want something to get caught in there, seize the wheel, and cause you to flip over.
If you have caliper brakes, then you'll want to make sure your tire isn't so big that you have to deflate it to get the wheel in and out. (Or, if you don't mind doing that, knock yourself out.
)
- Scott