I can try to hold a speed of up to 28 mph on crowded city commutes for as long as possible but without riding like a jerk, my average speed would only be 15 mph or less, or almost half my maximum sustainable speed!

If you're only averaging 11 mph, double that figure, and you could be hitting >20 mph in some segments. >20 mph is not bad at all on a hybrid bike with pannier. I'd even say you're doing fantastic!

So I'd say you're doing perfectly fine I suppose.....But if you say you're only averaging 11 mph in nearly empty flat roads far away from the city, they yes, you have plenty of room to improve on 11 mph speed.