I don't think it's dangerous as a general matter. I do it just to stay in practice, and have my hands ready to grab the bars in a fraction of a second. I need to be going not too slow, 12mph as someone said is about the lower limit, and not too fast as I don't want to lose control or crash.

A related practice I have been doing lately is to unclip one foot and bring it up and back behind my butt, grabbing it with the opposite hand, to stretch the leg. This is done with one hand on the bars. Saw some pros do this during a long race.