The variations in metabolism is a weird and wonderful thing. I hike, ski and snow shoe in the mountains of New Hampshire and ride a bike in Rhode Island. No matter the activity, it takes me a good 15 to 20 minutes to thououghly warm up. Since I already know this, I always startout at a comfortable, easy pace and ramp up slowly. Some of my hiking/backpacking friends, on winter trips, in temps in the low 20s F will hike in shorts and within 10 minutes begin to unzip outer layers. I have actually seen a hiker in temps in the high teens drenched in sweat with no shirt. There is simply a vast spectrum of people with huge physical abilities and requirements.
On the cycling front, I also warm up very slowly but I did read about a pro cyclist preparing for a time trial warming up for 20 minutes. I seem to remember it was Chris Horner somewhere or other.
I go with my instincts about what feels right or healthy. If you warm up slowly, that is OK - go with that.