You shouldn't be having much difficulty shifting down up front even under modest load. Check your low limit adjustment setting of the front derailleur. First confirm that the side plate of the derailleur cage furthest from the frame is parallel to the big chainring. Then shift into the low ring in the front and check the position of the inside side plate. When shifted into low (I'm assuming a double setup), the inside side plate of the cage should be about 1mm away from the edge of the inside chainring. It can get really close because the lowest gear combination, large sprocket in the back with small ring in the front, are pretty close to being aligned with each other which provides a straight chain run. Test it in this combination and if you're getting any rubbing on the inside cage plate, back the low limit screw out, counter clockwise 1/4 turn, to open up the gap just a bit.

Adjusting the low limit of the front derailleur has the added benefit of restraining the "leap" the chain has to make to drop down to the low ring. A shorter leap is less likely to result in a dropped chain.