Cold doesn't slow me much if I dress for it ... but I am loathe to ride in cold weather.
Heat ... when I was really fit heat was not a huge issue but it was an issue ... I felt the precursors of heat stroke after too hard an effort in high heat. But that took years of daily riding in 90+ weather. Lately I find extreme heat slows me considerably ... and again, being old, fat, and lazy, I loathe riding in extreme heat.