Fork might be slightly bent; frame may not be centered; tires not the best for riding no hands; headset loose; wheel not true; body not centered on bike; or you just need to practice more.