I need to exercise caution in such case, too (when passing a stopping/stopped bus on its left): For one, to make sure to watch for cars coming from behind, some of which do not slow down; also, there is a small chance the bus may not stop long after all (no/few passengers getting on) and could quickly leave the bus-stop while I have not completely passed it.
The first is the main worry. The second isn't a big risk, IMO - assuming you aren't pootling along at 8 mph, you should be well clear of the bus no mater how short the stop is.