The problem I think for many is the incongruity of the relational sizes within each nominal size. Eg. look at this chart and see how consistent these dimensions work for you.

For me, going down the chart, I'd be in a Large Chest, Medium Waist, (don't know hips), XL or XXL sleeve, Small inseam, and I'm XL in height. Generally speaking, I'm 185lbs and 6'2"