It won't make much difference for warmth, but I think the right size for a tent is N+!, where N is the number of occupants.

That extra room is really useful if it rains overnight (do you want to sleep on or under your gear bag(s)?). It also gives you a bit of extra room, useful for dressing or undressing in the tent. And if you're tall, it gives you room to stretch out without mosquitos biting your hands or bare feet through the tent wall.