I always use a baselayer. The only time I found a summer baselayer NOT to work, was when I visited Hawaii, and it was extremely high humidity.
But I also don't own any of those really light-weight, mesh summer jerseys. I find them too flimsy. So flimsy I don't even like putting stuff in the pockets.