Only time I ever wear a vest on the bike is when it's blistering cold out. I wear a down vest ($35 at Costco!) over my wind breaker. Keeps me warm without overheating me. Also squishes down in a stuff sack and fits in a bottle cage. Anyway, it does not flap. I suspect yours is too loose around the shoulders.