You should fit all bikes primarily to top tube length. You can raise a seat very easily but you can't easily raise your top tube. So if you have a longer inseam than most people you're height, its reasonable that you might be riding a medium as opposed to a large. But a small is likely, just way to small for you. If you're unable to sit and spin ... it's definitely way to small. AND ... the fact that you can't sit down on the frame size one smaller than yours ... tells me that you should be giving the large a serious try.
Seriously, the "nimbleness" of the bike has much more to do with fork geometry than the wheelbase.