Sure!

Andre Greipel, captain of Lotto Belisol, is 6'1" and rides a "Small" Ridley Noah Fast (54.5cm ETT). He's one of the strongest sprinters in the world, and for him the difference in stiffness between frame sizes is actually noticeable. By far the most popular reason for riding small is for the shorter head tube--some riders even choose track stems (as shown here) to get low enough. For a pro, position is optimized for covering a lot of miles as fast as possible. Even the less flexible guys (like the Schleck brothers) are more flexible than most of us.

It's very likely that the average rider will be slower over long distances on a bike with such an aggressive position.

Edit: None of this is to say that a 54cm will be too small for the OP--just adding some perspective.