Only remove what you know how to put back on.
Even a bad paint job looks good to others when you pass them on the road. They don't have time to see the details. Then again, even a rusty paint chipped bike still looks good going down the road. Again, I don't have time to focus on how immaculate your bike is.