I figure the tools based on the bike's need (what size what is used where). For even relatively short (12 miles and under) trips I like to be able to deal with:
a) one flat tire (tube, levers & pump)
b) saddle and seatpost adjustments (usu. 5mm & 6 mm allens)
c) brake adjustments (whatever it takes for pads & bike attachment & a screwdriver for the spring screw if there is one
d) rain, a plastic bag to protect my Brooks
e) complete breakdown, a lock, and a cell phone (& bus fare)


For longer rides I like to be able to deal with my stem, pedals, cleats, rack attachments (more allen sizes, wrench?) and a second flat (patch kit) and maybe even the chain (chain tool). I also throw in a couple zip ties, cuz people say they do alot in emergencies, tho I've never used 'em.