You can do both, but if you feel comfortable with not having quick releases go ahead and remove them.
I leave them on my road bike because those tires are more easily punctured vs MTB tires.