Personally, changing the fork would be my last modification.

- loosen your grip on the bar. Lay your hands on it, don't "grab" it.
- handlebar position
- handlebar (shape)
- grips
- tires
- tire pressure

would all come first