To answer all of the questions, no I don't travel very fast ATM because of said 'inbalance' but I went out again this morning and was feeling more stable.
Yes, I've had inner ear trouble since I was 9-10yo, I ruptured my ear drum and now get numerous inner ear infections and have 20% hearing loss in my left ear but I don't think that is the cause.
I haven't been on a bike in almost 20 years so I guess I'm kinda like Bambi just learning to stand, not very steady.
And lastly, my core is about as squishy as jelly, not very strong but I'm hoping to fix that. So, like bau said, I just need more saddle time and the balance will come with it.