main problem i have with frames that are too small is the more aggressive saddle/bar drop (due to shorter head tube). handling will be affected as you compensate to fit the frame (as carleton pointed out), but that's a secondary worry for me compared to neck/lower back pain from the low front end.