Clever
One solution I have used is to get a pair of cheap black athletic terry/elastic wrist sweat bands. I'll put one around my ankle and just leave it there. When it's time to ride just pull it up over the pants leg to hold it in place. When done, just pull the pant leg out and leave the band around your ankle. If you are wearing black socks nobody will notice it. Great for commuters or milk runs. You can get a pair of them at WallyWorld for about $3.