Where were you camped? Close to a town? Close to a bar, restaurant or some other indoor entertainment?
insipient cold usually comes from damp clothing. You found out this, and in cool to cold weather, it's best to stop riding, set up camp and change immediately into warm, dry clothing.
Warming the extremeties is difficult once they are cold. Usually, I will change into those dry clothes, then dive into the sleeping bag. Forget fires. I have rarely used one, and on one of the coldest rides I've done, a mate attempted for two hours to light a fire and keep it going. In the end, he gave up and did what I did, and got into his bag in his tent and warmed up.
I also like being able to cook from the door of the tent. The radiant heat helps (and isn't enough to endanger the tent), and food is a mighty big help when it's so cold. And yes! Energy bars are next to useless in such cold weather -- I had forgotten all about that, but I have experienced it, too.
It still might have been advisable to get going earlier. You might have got into camp earlier, and had the opportunity to get the firewood in daylight. I am not keen on arriving late into a campsite and having to set up in the dark. It makes you too prone to errors and losing stuff.
I mentioned the restaurant and stuff in my opening line because bicycle touring in those conditions isn't just about camping out. We are lucky here in Australia in that the campgrounds often have their own kitchens and entertainment/TV rooms with heating. I've often set off to town for a meal or at least to get provisions for such.
As valygrl highlights, a book, or at least a magazine, or a puzzle book is good to have. Plus a campchair converter for sleep pads... they might seem like excess weight, but I spent nigh on six months touring and working and camping, and it was a godsend. Plus, you can wrap yourself in the sleeping bag and have support for your back.
Another little addition that was really helpful for me was a piece of closed-cell foam sleep mat cut to about 1 foot by 8 inches. It's ultra light, can be stowed in a pannier, but it's stiff enough to act as a knee table for eating and supporting things like a writing pad, or puzzle book or whatever. It's also very handy as a bum pad when sitting on the ground or wet benches and such.
And, if you take a pen and paper, you can write stuff, like a journal. I know that with hi-tech stuff these days by which you can key in info, this sort of thing seems like a drag, but it certainly helps while away the hours.
This all probably won't be of use to you now, but might be to others. I hope so.