In the city I work in there are typically multiple lanes going in any given direction so people can pass me by changing lanes. Also when traffic is at it's worst I can move as quickly as the cars. I don't have much for hills to worry about downtown though.
I'd not be comfortable either if I was going 5 mph uphill with a bunch of angry drivers behind me. If there wasn't room for me to stay far enough to the right for them to pass, I'd be looking for another way home.