Unless I'm out with my wife and kids, I'm always in "full kit".
It took me a long time to finally try specific pro cycling clothing, and once I did, I found it the single most important factor in cycling, closely followed by nutrition.
It depends, of course, on the type of riding one does.
I can imagine that for more relaxed leasure rides the clothing doesn't matter much, but for more serious riding, where one pushes him/herself towards the limit ... the imortance of pro clothing becomes increasingly more important.