Yesterday I experienced this same problem with 2 layers of wool and softshell jacket. At 29F, comfortable until about 1.5hrs. After re-reading some very old posts in this winter forum. I found the people who rode all day and said they were comfortable were wearing similar layers.
I'll try eatting after 1hr and then report back. So I'm wondering if the chilliness has to do with the body running out of energy. Maybe a precurser to a bonk?