I'd say that clothing will really matter once you ride for longer distances and under wider weather conditions. So, the ability to wick moisture away from your upper body and shorts area counts a lot.
Cycling shorts hold things where they belong and give some moisture/chaffing protection.
For colder weather, it's nice to have a wind break layer on the outside. Look for jackets that have long enough arms and tails for bike riding. Also, fit begins to matter at speed, because loose clothes flap around.
Finally, I'm a big fan of wool socks, even thin ones. They're expensive, but comfortable even when wet. In the winter, they're the cat's cheese...whatever that might mean.