It all depends.
Never leave anything you can't afford to lose on your bike.
So... if you leave panniers, then remove anything of significant value (laptop, cell phone, wallet, etc).
Around here, if you're out shopping, it is usually ok to leave a trailer, panniers, etc outside (without valuables). I'm not sure I'd park a bike on campus all day with them.
Maybe a steel basket, if that fits your bike and personality?