You described it a bit awkwardly, but no, you are not wrong. That is exactly what happens in terms of saddle-to-bar drop (or rise).

The top tube also gets longer on bigger frames, though, which can stretch you out.