I tend to agree that extra stuff is often detrimental to the level of comfort and I don't get when folks say they carry 60 or more pounds of stuff "to be comfortable". To me there are two kinds of comfort that are important when touring and they are comfort on the bike and comfort while sleeping. Comfort on the bike is best achieved by leaving stuff home and carrying a lighter load. Comfort while sleeping for me requires a pillow and a sleeping pad. Fortunately the two types of comfort are not too badly at odds with each other and I can get my gear load down to 25-30 pounds and still have good sleeping comfort. For me that seems to be the sweet spot for gear weight.

That said if you sleep well without a pillow by all means leave it home.