That seam is likely a major contributing factor to your issues. There is a reason why roadies wear that spandex.

I used to commute home from working all day in a sweat box of a building (warehousing/distribution) and the bike part of it was anywhere from 8 to 14 miles depending on route. I would wear 'tighty whiteys" under denim shorts and I was miserable. I then started wearing cheap compression shorts under the denim and that helped a bit- mainly due to moisture management (and helped prevent chafing/thigh rub). But I got tired of my shorts literally soaked with sweat by the time I got to the pick-up point after the ride (where I would toss my bike in the back of the wife's SUV for the ride home). Then I decided to try out a pair of cheap moisture wicking gym shorts (Starter w/Dri-Star from Wal-Mart) and that made a huge difference in ride comfort.

Something else to consider is the type of underwear you're wearing. Boxers are the worst, with boxer briefs being a lot better, IMO. Basically, a bit of compression down there is actually good, the dangling bits flopping about isn't.