If your rear-wheel is drifting to the side, that means you're leaning your body to the side, like leaning on a wall or doorway. Practice using your elbow to lean on the wall and gradually put less and less of your weight on the elbow. Then extend the elbow and gradually push yourself further and further away from teh wall. This requires that you put all of the weight on your seat & handlebars. You'll find that the rear-wheel will no longer drift.
Also requires a balanced touch on the handlebars where neither hand has dominance. A lot of of people grip their bars and pull in order to steady their upper-body against uneven pedaling-strokes. This causes you to be wobbly and the front-end to wag back & forth. Instead, curl up your fingers under your palms and place them on top of the bars so that you don't grip the bars. Your weight shoudl be on heel of the palm and your fingernails should be resting on top as well.