Both for me. Try sleeping next to a lake at that temperature, and I'm sure it'll happen to your bag too. It's not guaranteed that you'll notice though. It's almost impossible to tell if it's happening unless the shell is translucent enough to see the down sticking to the underside.

This is where a VBL can really help. A couple pack liner bags can do the trick. Rip holes into one to make a sleeveless shirt and pull the other one over your legs. I've used this to thoroughly dry out my bag with great success.