I wouldn't necessarily say it's the fact that it's a fixed gear. There's a zillion variables.. minute differences in geometry/weight distribution, different wheel weights (heavy rim = greater angular momentum = they 'want' to stay upright), different tires, etc, etc, etc.

Then again, I've never ridden rollers at all, let alone being able to distinguish between riding them fixed v. free.