Is this a new install or a new cable? If so, maybe you didn't ensure the shifter was in the relaxed position by operating the small lever a handful of times before you hooked the cable up to the derailleur. Detach the cable from the derailleur, pull some tension on the cable (by the downtube) while operating the small lever until the cable doesn't move anymore. Re-attach the cable to the derailleur and retune the der.