While I personally would never wear it (nor any predominantly yellow jersey), and I'd probably snicker at someone I saw wearing one while riding . . .if you want to, by all means please do. I'm guessing there's little chance you'll be confused with any actual Yellow Jersey winner so it's not really a big deal.