I like Voler race cut in XS for a tight aero fit, and I am bigger at 5' 6" 130 pounds. Depends on how loose you like your jerseys, but small might be a little loose? Small in Voler and Pactimo fit me like what regular people like for their jerseys, but I like them tighter.

Edited to add: iirc, xs voler race fit measures 17-18 inches across armpit to armpit and 21 inches from bottom of front collar to bottom front hem. check against your current jersey?