Bigger frame = More stable, more comfort. Good for straight-line riding.

Smaller frame = More "flick-able", meaning the frame is easier to throw around while climbing or sprinting. Good for racing and climbing.

I would decide which is more important to you, and then choose accordingly.