You might also try upright handlebars. The shape of your bars can drastically change your riding position, how the bike feels, and how it fits you.
​​​Unfortunately there are a jillion varieties of upright, northroad and mustache-ish bars, but it's hard to go wrong with something from the northroad family of bars (Wald 8095s are less than $20).
​Anyway, definitely adjust your seat/bars/whatever so you're putting less weight on the parts that hurt after a ride.