It would kind of bother me, but fixing it depends on whether or not I would think that I could cleanly remove it without destroying the badge or marring the paint. Then I don't know what I'd stick it back on with.
But... it does make your bike unique.