The Tarmac/Allez geometry runs a little on the big side for the frame sizes. If you have a 130mm stem on your bike though, I'd go up a size. Stem length does have an effect on handling.

Sometimes I'm measured to be a 56, but I know I prefer a 58 with ~120mm stem. It's my preference to be stretched a little bit, and maybe it's yours too.