It is less of an issue of pucker factor and more of an issue of where you ride.
I ride motorcycles. Accident research has suggested that 80 mph on a highway is safer than 30 mph on city streets.
Higher speeds would make certain trails, residential streets and sidewalks unsafe due to dogs, children, walkers and other hazards.
OTOH, I am getting to speeds that are unsafe in these environments on my hybrid anyway so moving to a faster bike may not change the equations much. And I am hearing that the speed difference is not as great as I had previously thought it would be.