I take both. I get about 10 - 15 flats a year, and I ride less than 4,000 a year.
I like the pump to get a little air in the newly installed tube to make sure it's seated correctly, then I fill up with CO2. If I mess up with that (I usually have 2) then I always have my pump.