Bad idea. You will also lower the BB by 1" (25+ mm), which can lead to major pedal strike problems if you are riding fixed. It's the same as if you went from 165mm cranks to 190mm cranks. Frames that are properly designed for 650c wheels have less BB drop to avoid this problem due to the lower height of the wheels. Also, the bike will sit lower, because the fork will have shorter legs and you won't need to use super long reach brakes.