No matter what anybody says, size matters.

Now...regarding your fit...you need to ride it as is for a month before messing with things. You'll get a feel for how it fits your body with more miles. You'll also find your body will strengthen in certain ways to accomodate your setup. Neck, knee, back pain are all warning signs that the fit is off but until then, ride.