Frame geometry matters a lot more than frame size, in the sense that a 54 Madone may be too big and a 54 CAAD10 may be just right, or whatever....but the general rule of thumb is that it's easier to make a small frame bigger than a big frame smaller.
Don't go too small though.