I'm all about layering. I bought a non-bike specific jacket from ems.com that has armpit zips. It is actually described as a light insulator.
http://www.ems.com/catalog/product_d...82574488760917
I also have this guy from canari.
http://www.rei.com/outlet/product/766877
I plan on using the insulator inside the shell. Since both of them are somewhat stowable, I will shed one or the other.
If you want to have something for casual wear, why not pick up a polartec 200 fleece and use the above shell?? Once you get to your destination, pack the shell in your pocket.