Congrats on the PRs and Santa Barbara 100.
What I do is match my event goal duration / distance to my endurance training (zone 2). Generally, my events are short with the most being 40 km ITT or TTT. So if I am going to race a 40 km time trial, I need endurance in the time trial position for around an hour. Riding longer rides at least once or twice per week is beneficial. To have speed at 40KM I need fast shorter efforts.
In general, my training is two to three interval workouts per week with endurance on the weekends. Some of my racing friends, similar age are doing two interval sessions per DAY. One in the morning and another at the velodrome in the evening.