Smaller frame: Smaller, stronger/stiffer (at least in theory), lighter, twitchier ride, maybe a bit more road buzz.
Larger frame: more comfortable ride.