That's pretty much it. Maybe your coworkers are thinking of going faster downhill instead of climbing (or maybe they're yanking your chain ).

To expand on this --

For short sprints, like the velodrome racing around a tight banked oval like they do in the Olympics, anaerobic, high-power muscle effort is good, so you could use a "taller" gear combo that's harder to pedal but faster overall. But, riding for 2-3+ hours at a time should be more of an aerobic activity -- your heart & lungs should be handling the real workload, not your leg muscles. If you're forced to push too hard on the pedals, you'll fry your legs, and you won't make it up the climb at all. Even pros might use compact 50/34 cranks depending on the terrain, so there's no reason for any of us to shy away from them.