Easier than that, 1/8" drill bit will do the trick, right through the middle of the rivet. That's what I used to remove 1000s of rivets when I restored vintage race cars. If you want a lot of speed, use a drill that runs off of an air compressor.

You could start with a smaller diameter if you think the 1/8 is too big.