As you increase tire size (all other things being equal) you increase the amount of weight. The weight in and of itself isn't a lot, but it's rotational weight so it very much affects acceleration/spin-up. Not so important if you're commuting or touring but if you're racing, the extra weight equates to a lot of extra wasted energy over the course of a long race.
Also, it's not as simple as saying that a wider tire has less rolling resistance, it really depends on the surface. A super skinny tire inflated to max pressure has the lowest rolling resistance if the riding surface is exceptionally smooth (track racing or insanely well maintained roads and bike paths). The optimal tire really depends on where/what surface you ride on.