I can tell the difference between 80 psi and 100 psi in my tires
The newer mid to high level road tires and rims can easily handle 120 psi.
Another way to make your wheels faster is to pump them up with nitrogen.
On the other hand, if mountain biking on dirt trails; climbing can be made easier with lower pressure in the tires.