then i would go with two velcro straps...each long enough to do at least 1-1/2 loops around the ankle and calf. If you can, rubberize part of the calf strip so it doesnt slide around.
either that or get a neoprene ankle warmer and wear it on that leg, and just deal with removing a shoe, sliding it off, and rolling your pant leg down...