Make sure the velcro is to the outside along with the pant cuff. I have a cheap $3 velcro strap from the bike store and it even holds my prosthetic leg in my pants when it falls off.
You could try a collar like for a cat or a small dog. It may look funny but I bet it would hold well.