I'm personally terrified of rideing on the sidewalk. I even avoid it when the sigh says I'm allowed. Sidewalks are slow and dangerous. What screws me up the most is when the sidewalk/bicycle path end and you have to somehow make the merge back into traffic. If I was on the road the whole time, most of the drivers will have known I'm there. Leaving the sidewalk and joining traffic has got to startle those already half panicked motorists a little more. At least that's my safety perspective.
Mostly, sidewalks are just annoyingly bumpy on my roadie tires and someone is always in the way whether it's dumbass minivan or dumass ped with headphones cranked.