I have that jacket, and it runs very snug: I'm normally a Large in Rapha tops, and the XL jacket is very snug: I can wear it over a thin baselayer in temps down to about 45, and over a thick baselayer down to about 40 degrees. No way could I wear it over a baselayer and a jersey, but I don't think that's the intended use for it anyway. It's designed to be a rather minimalist and snug outer layer for cool/cold weather. I just think of it as a very warm and somewhat wind resistant jersey.

By the way, my chest size is the same as yours: about 38" currently, though my shoulders are rather broad. I couldn't even fit into the Large jacket, and find it hilarious that they recommend a Medium for my chest size.

tl;dr: size up.