I was going to post something saying that you're not going to get germs from a used jersey, so if you think it's gross, it's just psychological -- get over it.
But then I realized that the same argument could be made about used shorts (or underwear), and I think that's gross.