Don't worry about how it looks! If it feels good, do it.
I went to a slightly larger frame on my last bike (per Rivendell sizing, though it's a Calfee and not a Rivendell) and I'm much for comfortable overall, plus . . . I ride in the drops a lot now, as to almost never on my smaller framed bike.
So don't become too focused on one part (like the stem, or the drop depth of the bars), but look at the whole picture (which you probably knew anyway . . . ).
Rick / OCRR