I think it's been said well but I doubt you'd see any meaningfull gains in speed with those clothing upgrades, but perhaps some in comfort. Human minds plays tricks on us though, and sometimes doing this does make us faster, but its just a temporary mind trick. First time I put on a full bike color matching kit I felt fast, and went fast. But now-a-days I ride more like you and don't have a problem. A $10 (or less) pair of poly athletic underwear prevent most any chaffing and can be worn under any shorts. I have no problem riding barefoot on platform pedals either, done century ride and multi-day tour approaching century distance daily without needing any of this stuff. But I am not breaking any speed records though. I do prefer the bike clothing on a really hot sweaty ride though, as they mitigate sweat a little better and it doesn't show, my regular shorts I typically wear are actually mostly cotton and really show the sweat.

If I were in your shoes as they say, I'd skip the accessory gear and focus on gears [literally] if you're on a SS and in the hills I would think having gears would be the single biggest gain in efficiency, followed next by lighter wheels (unless your are already light).

You might have missed he said that he was on a single speed, 48-16.