Another way to look at it is that a smaller frame has a shorter top tube,

but the front wheel can only be so close to the BB.