I think it depends on the road conditions.

If you can't reach a very high top speed (i.e. you ride around in the city, up hills etc.) then I don't think it would make a big difference.

If you have long stretches where you can hit 18-20mph, then I think it does; this is just based on personal experience.

I have a short commute, but I wear what are essentially swim-shorts (non-baggy, stretchy polyester material that dries fast) and a compression shirt that feels like a similar material. I have two sets that I alternate between and just hand-wash the shirt and line-dry when I get home each night.