IMO, you don't have to prime all the time. I usually will on something that I want to look good. If nothing else it gives a chance to see where surface imperfections are that might need some more attention. Those sometimes are not so easy to fix if you just spray the color coat. High build sandable primers are great for this if the paint you are spraying on them is compatible.

I'd clear coat anything that I wanted to protect the color coat from getting scratched to the primer. Easier to fix a scratch in the clear than a scratch in the color...... usually. As in everything, it depends on what paint and primers you use. And how bad you want to make it perfect.