I started riding two years ago and bought a road and mountain bike. Both are a blast as far as I am concerned. I do more road but when I get to the single track on the mtb I revert to a kid every time
If I lived right at a single track or within riding distance I would maybe do more mtb

So no you are not crazy. Go for it and have fun and be careful