That type of jacket isn't meant to fit all that snugly. It's more of a shell to add layers underneath. A smaller size may not fit your shoulders.

If you want an aero jacket, go for a different style.