I'll pack up to 50 lbs. total in my 2 rear baskets and the only drawbacks are-
1. sluggish acceleration
2. Handling/turning suffers at VERY slow speeds, but that's true without carrying a load. You just have to anticipate a bit more if heavily loaded.
I would expect a heavily load front basket to be much worse in this case.

It probably makes a difference how much YOU weigh in relation to the load.