When you say 'wobble', do you mean a weird vibration like your car has a tire out of balance, or you just can't hold a straight line? If the latter, ease off on the grip. You should be 'steering with your elbows'; at least that's how I was taught.

Only thing I can think of for the former is the headset is loose. Maybe the geometry has something to do with it? I've cruised downhill in the aerobars at 45mph with no wobble. Scared the s**t out of a race marshall when I went zinging by him on his scooter!

Oh, and everything 900aero said.