Depends on what you mean by "matters". If you are 00:00:01 out of first place on the sprint to the finish in the Tour de France, everything matters. If you're an average Joe enjoying a recreational ride through the countryside, then the minuscule difference between a 19# and a 21# bike probably won't mean diddly.