Take the windbreaker with you. Assuming the windbreaker has a full length zipper wearing it fully open is not that big of a deal, people will think a superhero donning a cape just dropped them ;-).
If no zipper you can always wrap it around the top tube or seatpost and use the sleeves or a bungie cord to tie it up. Make sure it is secure and double check during rest stops it is safely tied.
Good luck and ride report when you can