Both. Get a good load of groceries in some panniers and try riding into a headwind, like you did.

Weight makes the bike handle a little bit different, but unless you've got some pretty stiff hills, you don't notice the weight once you get rolling.

Wind you're going to notice, period. And suffer. And despite what you might have read or heard, there's no such thing as a tailwind.