As others have said - Take the pannier, leave the rack.

Not sure what bus/transit company it is, but I know some have rules about that. I think that the commute bus I use sometimes does not allow you to have bags attached to your bike while on the rack. Not sure if it impacts fitting another bike on, or if they just have it for liability (someone complaining about liability for it falling off, etc.).