You know those chin straps are adjustable, right? And most helmets have some sort of way to adjust the fit (dial, pad thicknesses, etc.)

I'll admit I have to spend 30 seconds readjusting the fit of my helmet when I switch from riding without a hat to riding with one. Doesn't bother me. (I use a powerstretch fleece hat... almost overkill)