I would think a cap gets mighty warm when it's still 60F out? In such a case, I'd try foam or rubber hearing protectors. That will prevent the air from entering the canal. For some extra protection during colder temperatures , add thin headband or the previously suggested polyptopylene balaclava which will not interfere greatly with the helmet fit.