I always use the smaller panniers for up front and the bigger ones in back, but that doesn't mean I can't still put more weight up front. I usually try to put more weight on the front as to balance out the bike, and put lighter, but bulkier items like sleeping bags and clothes in the rear panniers while putting heavier stuff like tools, water, toiletries-bag up front. If you put too big of panniers up front, I fear it may interfere with steering, like dragging on the ground through sharp turns.
I did do one overnight tour once where I just used my two front panniers for everything and strapped my sleeping bag on the back rack, and this set-up worked really well.