I've ridden gravel around the area on a road bike with 25 mm tires. It is not optimal and slow going at times but doable. Brushy Mountain Rd over the top of the tunnel is fine gravel over packed dirt and will be a pleasure to ride on a road bike but this unnamed road is a bit gnarly.

You could do it if you go slow and are prepared to walk the steep sections. There is one very steep spot where I spin 35 mm tires when riding away from the trail.

It is worth taking the foot path away from the trail and turning right at the gravel road to check out Racoon Creek. Very pretty back there.