If you're just talking about cool spring and fall riding, I have a light wind breaker jacket I use for that. It compresses nicely and will fit in my jersey pocket when I need to take it off. This works well for me in changeable weather. I'd suggest one of these. You can find them cheap on Nashbar etc. I can't see a situation where I'd just want a vest. Nor would I want to fiddle with detachable sleeves.

Just my own take. YMMV.