You could just layer a fleece and wicking layer underneath. The jacket will serve its role as a wind/water barrier while you could use something else to insulate.