I think I may have found a contributing factor to the problem. I measured my sit bones and the specialized chart suggests:
If your sitbones are between 70 & 100mm, you should ride a 130mm saddle, unless you ride in an upright, comfort position. Then you should ride a 143.

The saddle on my bike is 150mm, and the road bike has a 130mm saddle so perhaps this causes me to slip forward a bit as my sit bones aren't being supported as much as necessary. I think I will get a cheap 130mm saddle and see if it helps.