at 5'10" you should be right about in the 17.5 - 18.5 size, depending on torso lenght. the 17.5 might be a tad too small for you, but then again, trek tends to have a longer top tube. when you sit on the bike, you shouldn't be stretched out. Often companies put riduculously large stems on the bikes, you put a shorter stem on and completely fix the problem. also you shouldn't feel all bunched up, like you are riding a kids bmx bike. slight bend in the knees when your foot is at the lowest point in pedalstroke, slight bend in elbow, not hunched over and not sitting straight upright.
these are all general guidelines. if it's comfortable for you, that is all that matters.
at least ride it around and hop some curbs. if you can, rent one and take it out on the trails you are going to ride.
the only way to really know if the bike is comfortable for you is to actually take it on the trails you are going to ride. some shops have a rental program where the rental fee goes towards the purchase of the bike.