I've ridden my road bike on trails, grass, etc. and never had a problem. A reasonably strong set of wheels will hold up to almost everything one could throw at them.
If a bike's too fragile to handle the occasional off road foray, then how well will it hold up hitting a pothole, rock or whatever at 24 mph? Don't worry...most modern frames will hold up better than the rider off road.