Definitely try it out. There are so many variables, it's impossible to predict how your bike would handle. If the load is light enough to fit in two panniers, it probably doesn't really matter much where you put it.

I just finished a tour with only two lightly loaded panniers which I mounted on a rear rack and I barely noticed the difference between loaded and unloaded, as far as handling goes. I could ride no-handed at single-digit speeds, just like without the load. I'd only notice the weight on a steep climb.

Try riding in a stiff crosswind before making your final decision. I met a guy who was ditching his front panniers in Montana because of a perceived handling issue.