A couple times is nothing. It's much simpler to just put up with it and learn to enjoy it.
I was doing the same and felt I needed to gear up. Turns out those few times were when I had a decent tail wind. I just learned to enjoy it, like "hey, I'm going fast and don't have to bust my butt".
Otherwise, I'd have a cog that seldom got used

When you gear up, you lose something on the other end. Sometimes that loss is worse than whatever you gain on the top end.
I'm a "gear head" and make up different cog sets that fit my riding conditions. This has taught me that sometimes the grass just seems greener on the other side. I only ride flat land, so I was able to gear with a lot of closely spaced, smaller cogs. If I have an unexpected steep hill, I'm screwed.
My cogs work great for me, but like I say, it only works well for the flats.

Now an option would be to pick up a complete spare rear wheel with a different (same NUMBER of cogs) gear set off of Craigslist or similar. You can usually find one if you're patient. Then you could just swap wheels.