Smaller roller diameters correlate to higher rolling resistance. Which depending on your tire and riding style could mean more wear.

However with a fixed gear....I'd wonder about how good a workout you can get with larger rollers. Even my Performance Travel Trac rollers (85mm diameter), I need a medium high gear to get a workout (50x19+)...The Performance are frequently on wicked sale too (I landed mine for $100 or so).