Bah! Some cycle tourists pull trailers for thousands of miles. Maybe stuff will wear quicker, but so what? At least you are out there riding and enjoying it, and I bet your kids enjoy it too. Wearing down parts from heavy use is preferable to having a bike that is in perfect condition because it never gets ridden. Just keep the bike properly maintained and you should be fine.