Yes and no. Bigger tent = more surface area through which to lose heat. However, you can't really rely on your tent to keep you warm, because you need ventilation to avoid condensation building up and soaking your sleeping bag. Condensation is caused by two things: water vapor in your breath, and dropping temperatures reducing the amount of moisture the air can hold. Smaller tent = more concentration of moisture in the air, which leads to more condensation. Condensation is prevented by ventilation (cycling the moisture in the air out of the tent), but that basically means you are letting in the cold. So that means you need to rely on your sleep system (mattress/pad, sleeping bag, thermal underwear, cap) to keep you warm instead of the tent. In practice, you'll find a balance of ventilation that avoids the condensation, but some warmth is retained, but you're only going to have a 5° to 10° difference from the outside air.

Have a good sleep system appropriate for the temperature, and eat a big meal (carbs) before bed, and you'll be fine. 40°F isn't that cold.