yes indeedy that would suck. i would think that you have a link or two left over from when you shorten a new chain to put on your bike, throw that in your tool kit and a cheap chain tool and your off. that said i don't carry one around when i'm riding in the city; it's easier to show up at keith's with food/beer and borrow one of his!
out in the country i'm super paranoid (don't know about the rest of you) so i end up carrying just about everything, even a spare tire (which these days is around my waist).