To get into the correct position, you need to move your saddle forward. This effectively shortens your top tube. Shorter top tube = smaller number next to the frame size.

It should make sense that a bike you ride with your hands would be a different size than a bike you ride with your elbows/forearms.