I've never taken the convertible sleeves off mine, but I don't wear it often either. It's a Gore something or other and it's pretty damn tight, apparently I needed the next size up. Oops.

The sleeves are a bit bulky for storage too, after you take them off, so that kind of negates the advantage. I usually prefer a wind vest & arm warmers down to about 40 degrees. The vest and arm warmers both wad up pretty small.