They're not really rivets! [Okay, well, some are]

This comes up periodically, but head badges are traditionally attached with something called a "drive screw", which you can think of as a one-sided rivet, if it helps. I remember growing up working in Grandpa's wood shop, and he used fluted nails for the same reason: it helps hold two layers together (better than just pounding it together with a straight or tapered one), while also being very quick to install.

They can be popped out from the back if the heads are solid, otherwise (as I believe is actually the case with some Raleighs), use a slightly-larger-than-the-hole-in-the-rivet twist drill bit, and go through fast in one operation, or you can wind up grabbing and spinning the rivet, rather than drilling it.