32 pounds is a pretty good load to put at the back, but I've managed loads that are similar without much drama. (How much can a bushel of apples weigh, anyhow?) A couple of thoughts:

Wind can make just about any bike feel squirrely, especially at high speeds (30 mph plus, IME). Headwinds turn into crosswinds going into turns, or in a ripple in the mountain, and if you're not on top of the situation, it gets interesting fast. Passing traffic can also blow you around. If you've got drop bars, this is the time to use them!

If you start feeling a shimmy, skip the front brake (unless you're executing a panic stop). That rear load is going to keep your back tire on the ground, so rear braking will be effective. Think of an 18 wheeler, using trailer brakes to help stop, or a car pulling a trailer that tries to fishtail under hard braking. If the braking is in the back, it's not going to be dancing around as you slow or stop.