I’ve done this but had a threaded eyelet located at the seat stay stripped after a while so stopped taking off the rear rack.