Nice bike. If the 54 feels too large, there is nothing wrong with trying the next smaller size. Specialized road bikes run a bit larger for their nominal size, and their size recommendations tend to run large, too.

Personally, I'd rather make a bike that's a bit small fit larger with a longer stem and more saddle setback than make a bike that's too large fit smaller by moving the saddle forward and using a shorter stem. You'll have more options and feel more in control with the smaller frame.