Another idea that may help is a light weight base layer. I have started wearing a base layer under my jersey, often even on the more mild days. It seems to get more wet then my jersey, but the really light weight base layers also seem to dry much faster then the jersey. They are also small and light so you could bring an extra with you in a jersey pocket if needed.

Curious why you won't consider a bag of some sort - pannier or a backpack? It would let you carry spare clothes, etc and probably have minimal impact on your overall speed.