Good fenders, properly mounted, are a real PITA. Mine are mounted more-or-less permanently and only come off in extrema. So I voted "leave them on".
Bad fenders are available which are easy to install and remove. That is good, because you can't wait to remove bad fenders after you've installed them. So you've been using bad fenders, you should probably take them off at every opportunity.
HTH!