You can easily do some light off roading with a road bike, they aren't that fragile.

I had a similar experience, except even more extreme. I went from a 40lb+ wide, knobby tired suspension mountain bike with flat bars and upright riding position, to a 20lb 105 road bike with 23mm slicks at double the PSI. Way more aero, lighter, faster, more responsive, love the drop bars and brifters too. Super convenient being able to shift and brake at the same time.