I ride a 56cm Roubaix. I also ride a 55cm Jamis Aurora. And I have test ridden a 54cm Roubaix. They all fit fine. Just watch for the effective top tube length and head tube height.
You don't want to end up with a goofy stem to compensate for the reduced head tube height in the 54.