Either your wheel is out of true (in which case it would be noticeably wobbly at any speed) or...
You've had a run-in with "shimmy", and dang is it scary
Here's Sheldon Brown's explanation, with a link to a longer article on it:
http://sheldonbrown.com/gloss_sa-o.html#shimmy