Weight only matters when accelerating or climbing, really. When you're at speed, it doesn't matter nearly as much. It takes a lot of weight to matter. Maybe 40lbs will do it. But it'll make your ride suck, because the vest is going to block the evaporative cooling you get from moving at a decent speed. Ride faster, or find hills to go up.