Unless you plan on changing your stem to a much longer or taller one, that's well over 3" of unnecessary housing leading to the "inelegant" look you're trying to avoid.

​​​​​​Now if it's a touring bike, and you plan to dry your wet socks on the brake housing, maybe taller is better.