My helmet feels weird when I don't have a cap under it.
Suggestions:
Visor down only when sun is in front and getting low, or when necessary to keep sheets of rain out of eyes; your helmet is a visor of sorts when the sun is more directly overhead.
Position cap further down on forehead (nearer to eyebrows) to allow visor up without helmet conflict. (Cap positioned to accommodate helmet, not the other way around.)
Try different brands of caps; I've found some to be more helmet-friendly than others, largely based on size of, and material used in, visor.
__________________
A race bike in any era is a highly personal choice that at its "best" balances the requirements of fit, weight, handling, durability and cost tempered by the willingness to toss it and oneself down the pavement at considerable speed. ~Bandera