I'm kind of in the same boat as you, I'm getting back into biking after about 15 years of no riding, Bikes have got better and trails harder. the sport has gone a long way in that time, for the better.
One ride with a group on "intermediate" trails and well, same experience as you, Hang on for dear life, couple of bits I had to walk down and some, If I could have stopped in time, I would have walked down.
What came out of it all is the longer steam I'd put on the bike because I felt it was more comfortable was well wrong, would have never learned that by myself.
Ride with a group and learn from them, and it's a whole lot of fun.
PLAY, spend the time playing, bunny hopping, up and down curbs etc. do what's fun for you and enjoy it!