I always thought it was the imperfections of the road & planet Earth. A bike should roll perfectly straight when the cyclist is rolling the bike slowly on a perfectly level walkway or floor, handsfree from the handlebars. I usually try this with one hand on the seat and the bike & wheels as perfectly in alignment front to rear as possible. It's difficult to do because the front tire will want to come out of that perfect alignment after a few inches or a foot or so into the roll. I presume you could roll it backwards and the front wheel would stay in alignment longer as it's dragging/trailing instead of being the leading line ? Even on as perfectly level a sidewalk as one can find, there are still going to be imperfections. Even the tires have imperfections that may influence perfectly straight tracking. Controlling all of those variables is difficult if near impossible. Again locking the front fork a perfect 90 degrees with the vertical plane front to rear of the frame, the bike should roll perfectly straight.