If the sweater doesn't quite fit right while riding, you can always cut it up and sew a balaclava or a cap out of it.