I agree with your preference of the 2X over the 1X, given your planned use. The size choice isn't as clear, especially since you have comfortably ridden the larger size. The 51 comes with a 70mm stem so you do have the ability to go another 20-30mm w/o screwing up the handling. As pointed out above, there's not much you can do to make the 54 fit smaller. Nice choice of bike, wish my LBS carried Jamis.