Slip your arms out of your jersey, but not over your head (just over your shoulders).

You don't even have to take your helmet off.

Full zip jersey is even less of a bother.

Costs a few seconds of time. Worth every bit of it for the superior comfort of bibs.

I don't see a problem.