I have a Phantom jacket. It was tight when I first bought it, I hung it in the closet for 6 months and by then I'd done enough riding and shed enough flab that it fit fine. I have about a 42" chest and I'm wearing the large, often over a base layer + a jersey. So if you're really 38-39 I'm guessing medium should be fine. Though as someone else pointed out, maybe some models run tighter.

BTW the Phantom is a great jacket, I wear it through the California winter as a long sleeve windbreaker, and in the summer on foggy and windy days I zip off the sleeves and keep wearing it with just a base layer.