It could be a couple of different things but it sounds like the cable is stuck some place and while you are applying force (shift) it moves and shifts and when you move your hand from the shifter it tends to return back and reshifts to the original gear. I'd change the cable and the housing first (could be catching in some part of the housing that is torn apart on the inside). If that does not work, I will check the springs in the shifter. I assume that you have problem with only the rear or only the front and you are using friction shifting. If that does not work, then you'd need to adjust the RD or FD