The best thing to do is to try both at an LBS, I'm basically down to full time cruising in my 20" so I went to the shop in town and tried a 24" to see if it was any better and I hated it. If you go with a 20" get your seat post up and get a long TT. I'm 5"11 and I've got a 20.5, it's pretty decent, but I could definitely go for a 21 or 21.25 of money rained down from heaven.