If the distance is only 4 miles, just wear your work clothing, plus any overcoat, raincoat, etc. that weather may require. Unless you ride hard, you will not work up any more of a sweat than you would walking a mile or so.
I normally commute wearing a suit and tie. On hot days, I fold and carry the jacket and ride in my shirtsleeves. Alternatively, I leave my jacket at the office.
On the way home, you can hammer if you want to.
The kayak commute sounds great! When I was at the University of Washington, there were several kayak commuters, and I really envied them.
Paul