I honestly pay way more for clothing than I do on my bikes, relatively speaking.
I would never pay for Dura-Ace or Record, for example - but there's almost no limit to what I'll spend on clothing. Ultegra doesn't detract from my riding at all, but being cold/wet/hot does.
That said, I'm a big fan of Castelli summer gear. Their jersey and bib shorts suit me well. For cold weather gear, I'm gradually moving to Rapha wool. It's very expensive, but it's worth it in my estimation. Then again my favourite gloves are $10 'Axiom' cheapies...