At the VeloSwap last Saturday I scored a high-zoot pair of aero-bars.
I was looking for some since I intend to try time-trialing next spring.

I haven't been on the bike or in the gym for two weeks due to upper
respiratory crud. I mounted the bars on the 'cross bike and tried an
easy ride today.

From my house I have ~200 yards 'til some downhill. This is >40 MPH
downhill so I was fortunate to catch on before the hill that the hill
wasn't going to be the place for the first try. Very fortunate.

After the hill I have some easy rollers and discover that I can stay
in the bike lane, but that really was the level of control that I had.
After an hour I still couldn't stay on a painted line. It is like riding
without handlebars. You have no leverage and all that is left is
balance. Considering how stable the 'cross bike is compared to the
road bike I now know that I made the right choice, but it was a
crap-shoot. I didn't know better.

Thanks for the first-hand observation. Keep us posted on what you learn.

Exactly why they are frowned on in group rides.

Don't give up of course. Like clipless pedals, or someone switching to friction shifting (!), it just takes patience as your body makes using them intuitive. They are very useful for going into the wind, pulling against as you push a big gear, or finding a new, relaxing position. Don't grip them tightly and steer with your fists....instead steer mostly through your butt on the saddle as most experienced riders do... inclining the bike. Put most of your pressure on the elbow pads-- a better place for leverage.

You may have your elbows too close together: better to start out with your elbows farther apart than a time trialers...most bars allow for width adjustment.

Soon enough you'll be steering around road debris and even going through turns down on your bars. Do give them a chance and you can add one more new thing to your cycling repertoire.

Like Grampy, I don't recommend them for riding in close groups or on steep, speedy hills. Hard to flick your steering or get to the brakes.

I reacted more worried than you after my first rides with aerobars, I thought I was free falling. As CC says so eloquently, you will get used to it. The balancing and steering will become instinctive with time and use. I now do centuries on aerobars, against the NW wind, on Wisconsin limestone trails with lots of potholes.
It will take a while to get that level of control.

Like others, I've used aero bars for a while. As Will and CC have pointed out it just takes a little time to gain confidence in your skills....stay with it and it will continue to improve. I still come out of the aeros on fast downhills, though. I also do not ride with anyone while using aeros-except for a buddy where we both have our TT bikes and we will go out and do intervals together.

BTW-I now have bar-end shifters on my bars and really like the ability to trim the gears while keeping my hands on the bars. Just another way to add to the riding pleasure!

Every bike I own has aero bars. They take some time to get used to but I use them 50% of the time when riding now.

I had no intention of giving up. I was just very surprised and
wanted to help others avoid surprise. Somehow I am always
disappointed when you actually have to be competent!!

After ~100 yards I spread the elbow pads as wide as they
would go!

Since my speedo wasn't mounted where I could see it and I
haven't downloaded the records I don't know how fast I was
but anecdotally it felt fast.