If you ride in a group, there is a social aspect to "dressing up nice". So beyond comfort, it might make you feel better and ride better.

I am bad to neglect fashion when I am out running errands that don't involve a lot of human interaction. My daughter describes my attire as "an odd mix of post apocalyptic and neo homeless"