Without seeing the bike and how you ride I can't answer. It could be hub play at the bearing, axle flex, frame flex, spoke flex, or simply that your brake is too close to one side and you could solve the problem by moving it over slightly.
IMO with seeing it, you can safely rule out the skewer because that might cause the wheel to move, but it won't cause it to flex.
__________________
FB
Chain-L site
An ounce of diagnosis is worth a pound of cure.
Just because I'm tired of arguing, doesn't mean you're right.
“One accurate measurement is worth a thousand expert opinions” - Adm Grace Murray Hopper - USN
WARNING, I'm from New York. Thin skinned people should maintain safe distance.