I meant for "riding with people with lighter bikes and you're matching their surges" to generally include racing. Sorry if I didn't make that clear. I'm sure that it would fatigue you more. Other than that, even in a time trial, I don't think that you'd find the difference significant. Not having to match an acceleration (to draft for instance) you'd just accelerate for a fraction longer, lose a tiny amount of time and have a small amount of extra effort.