in my experience, the racks by themselves don't alter handling much at all. It all depends on what sort of load you're carrying.

A front rack loaded with just a rack bag makes it somewhat difficult for me to ride no-hands, as the bike gets increasingly twitchier, especially when I'm tired.

A rear rack with a rack bag doesn't affect my handling much at all, though it does perhaps make my bike sway a little more when I'm climbing out of the saddle. A rear rack bag also supposedly adds drag with crosswinds, but I don't have any of my VBQ's handy for reference.

A rear rack with panniers is definitely more difficult for me to climb on, and I mostly spin when my bike is loaded as such. Rear panniers also add more drag in headwinds.

A front rack with panniers also makes the bike a little harder to control, but the weight is mounted low, and centered, so the effect is not as pronounced as with rear panniers. You don't get heel strike issues with front panniers but they are a little trickier to load and unload if your bike does not have a kickstand.

So, in summary, it really depends on what you plan on carrying. Loaded touring? you'll proably want both anyway. Commuting with grocery errands? I'd go with a rear rack and panniers. Supported touring or randonneuring? If your fork is built for it, I'd give the front rack with handlebar bag combo a shot. Otherwise, I would actually go with a large saddlebag and some really small pouch/bag up front, like a Bento box and/or mapcase on the handlebars.