Couple suggestions. You may be overdressing. The first 5-10 minutes should feel chilly. You warm up after that and should be marginally comfortable.
The other is wear wicking material next to your body and something thinner per that to capture moisture and allow it to quickly evaporate. Flannel shirts and sweatpants act like sponges and just hold moisture