A smaller frame will have better handling (there is a margin of diminishing returns, and a margin of negative returns with this) but better handling means you will need to watch your foot/front wheel clearance a bit more often.

But if its not broken don't fix it.