I hear ya.
I think for me it's equally scary riding next to a bus or large vehicle. I've driven a large vehicle, and there's no chance of seeing a cyclist along side you. If a bus is moving and I'm along side, you'd see me finding the quickest way out of that position, normally I slow down.

At lights I stop right at the right rear corner, or directly infront of the bus so they know full well where I'm at. Unless I'm 100% I can get by the bus while it's stopped, I won't try and pass them, and I'll only go by on the left.