That's cold. Not even close to what you experienced. Just one time at Red Rock Canyon just north of Las Vegas it was high 20's to low 30's.

Some jackets act like wind breakers and when a rider goes 17 to 20 mph, that outer shell of the jacket starts to get cold and it transmits to the inner layers. Then its a battle between your own body temperature to keep the layers warm and the cold wind to work the opposite. Over a longer period of time, that cold wins.

I would imagine that second half of your 20 mile ride was the most difficult.