Or, more likely, the chainring is not centered quite right. There's usually a little play in the bolts that attach the chainring to the crank. See the section on "Centering Chainwheels" on this page:
http://www.sheldonbrown.com/singlespeed.html