I started this essay after learning of the kitty litter panniers and thinking, gosh, I wouldn't want to be riding that when a UPS truck goes by on a country road with no shoulder.
When you are cruising along on a bicycle, the force of drag is the main thing on which you are spending power. If your intent is to get a workout, this is not really a problem. But if your intent is transportation, you'd probably like to go faster and get there sooner. You probably would also like to carry enough stuff and be comfortable. To reduce aerodynamic drag, you want to make a smaller wake. There are two basic approaches. First is to clean up the shape, second is to make the shape smaller. The ideal shape is generally that of a fish, a bomb, a bird in a dive with its wings folded. You cannot get there from here with a normal bike. But you can avoid making it worse.
Bikes are not particularly aerodynamic and it's mostly your fault. When you look at the front profile of a bike, the rider is most of it. You want to look as small as possible. Road bikes really are better in this regard. Drop bars lean you down and bring in your elbows. So if you can stand it, do it. Don't wear floppy clothing if style does not demand it. Not only does it make you bigger but the fluttering also saps power. If you can feel anything flapping in the breeze, correct it.
Even without you, bikes are still not particularly aerodynamic. Bikes are made up of many cylinders sticking upright in the breeze like flagpoles - dozens if you count the spokes. There are excellent structural and monetary reasons for this. Solutions generally involve making things smaller, fewer, or sculpted. Hence 23mm tires on deep V rims with 18 flat carbon spokes, egg shaped down tubes, smooth webs between the tubes. You can take this as far as you like - people have preferences that should be respected. Some of these things reduce weight, some add it.
You don't want to add anything to the bike that further increases its drag. Since you are already creating a big messy wake with your body, the best place to put stuff is in that wake. In that light a big saddlebag or rack-top bag is better than panniers, although admittedly less useful, and as long as it's not wider than you. Backpacks are good in this regard too. Unloaded wire baskets cause less drag than panniers (just imagine holding either one against a river current). Folding baskets or fold-out panniers are a great idea - keep them out of the breeze unless needed. Directly ahead of you is also a good spot, since it isn't going to make the wake much bigger, so a small handlebar bag is ok. A big wicker basket is probably not great. Accessories like lights and bells add drag too. They are small enough and dead ahead of you so they probably don't have too bad an effect.
Once you have folded out your chosen cargo container you are going to be carrying a lot of weight and going slower and the aerodynamics won't matter as much. But still, the same idea applies to your load. Keep it shipshape. Tie plastic bags shut so they don't flutter. Zip it all closed. If the load is light and small enough, just use one side of your folding bags, or don't unzip the expanding section, or whichever.
Do I practice what I preach? Somewhat. My commuter bike is a mountain bike with slicks. I'm sort of stuck with it for now due to saving money, but its replacement will have drop bars. I do have a road bike but I'm not commuting on it because it's not modern. Its gears aren't low enough and I don't believe it will be reliable enough, and it's classic enough to keep it original. In this season I often wear normal pants to ride to work since it's cool enough to want legs covered, and I tie up the cuffs with a velcro strap. For the ride home I wear cycling shorts. I haven't ever switched from tee shirts to jerseys, mainly because I don't think I'm slim enough to look good in them. I have an expanding rack-top bag.