Right, the rubber does not hold the tire together; it's the underlying casing. If that's damaged it may be time to replace but if it's just the rubber, and you're not having any problems, you're fine.