Is adjusting your seat, steering & mirrors necessary when you drive? I know, bad example, but you get the point! Have it fitted to your body. It'll cost you less than sessions at the physiotherapist in a few years and you'll perform better during your rides.

One thing that you won't be able to adjust when using a smaller frame is the stack & the reach. Both are important. There's a few threads here on that.