I suspect it's normal for adults to feel leery of the skinny tires especially when used to looking at and riding 35c wide wheels.
You will get used to them and at the same time avoid (if you possibly can) gravel, sand, debris as they are not designed for that purpose. To put it another way, running out of pavement and hitting loose dirt, gravel or sand will likely not end well.
See if you can borrow a shorter stem from your LBS and try it out for a few days to see if it helps. I don't think changing out the stem will make it more twitchy, at least I have never found that to be the case?