yeah, i think your wheel's slipping in the dropouts. resistance pulling on the chain, moving the wheel forward, creating slack...but you said it goes away when you stop resisting?
and check out a store that has bmx parts, i think that's probably the best bet for short stack bolts/bmx bolts (i like to say bmx, it's easier to say). then again, most bike shops will probably have **** to fix singlespeeds...ah, whatever.