Your office mate is a doofus. I ride with a single 8-12 lb Ortlieb most days, to no ill-effect, whatsoever.

Two loaded panniers makes it much harder to get my bike down the aisle of the train without whacking folks, which creates it own hazards.

Frictionless roadbed - ouch.