Sidewalk riding is about 50/50... some places it's legal, other places it isn't.
Riding on the road is legal in about 95% of places you'd care about riding. You can't ride on restricted-access interstate highways, for example, or some local freeways.
So unless your altercation with the driver occurred on the I-95 interchange, he was wrong.