Does this bike have gears or is it a single speed?

If it does have gears, you may be starting out in too high a gear. It sounds like it's taking you a lot of effort just to get the bike moving, - hence the wobble.

All of what you said though would be typical of somebody just getting used to riding. Keep practicing. While some of the long technique descriptions above are probably accurate, I wouldn't worry about it too much. Trying to over think it won't help.