Update: I replaced the headset bearings and it was really easy to get it adjusted between having play and it feeling tight. I never even tightened enough to get it tight. After 60 miles today it feels perfect. I think I just killed the bearing by riding around so long with it too tight. Thanks to everyone for helping me understand how to properly adjust it.