I also got a PI PRO Barrier Lite vest and had the same problem. Fits good around the waist but the rest of it was baggy on me and frankly, it felt like I was wearing a glorified garbage bag. I got sick of it and didn't really ever wear it so I tried their Aero vest and it fits really good. Nice and snug but not too tight either and somewhat breathable.

I think the main problem with the Barrier Lite is that the material is fairly plastic like wear as the Aero is more of a fabric, but still blocks the wind well.