It may be that you are not shall we say, "compatible" with you seat. Wide nosed seats did that to me and changing to the narrower ones, i.e. Selle Italia Flite, was much better. You may also have the post too high and the rocking motion of the hips will add to the friction. But, as skitbraviking says, it could be your cloths. Experiment.
Dave