My experience is that as tire size goes up, you can handle more difficult terrain. For me, I'd use 32mm only if I were going to be riding significant gravel or if I were unable to avoid potholes. I find 28mm to be good for most urban conditions.

Generally, thinner feels faster and more fun to ride.

It also depend on you/your load's weight and your riding skill. Heavier rides usually use wider tires. More skillfull riders can manage bad road conditions even on thinner tires. Use your own judgement here.

as mgagnonlv said above, the actual width of the tire is what matters and the label may or may not match the actual width. Within a brand, the label is good for comparison though.