If you're really talking about drilling into a chunky stamped dropout (which would be a logical place if you're mounting a rear rack), I would generally not see it as a problem.