Wouldn't worry about compressing the bag with an extra layer on top. Jacket, towel, extra clothes, etc. Even if you lose a small amount of loft, overall you will be warmer. For me, it makes enough of a difference that I'm no longer cold, so all I can say is don't rule it out because someone says it won't help. Try it for yourself.