I ALWAYS lock my bike and I always use a U-lock. But hey i'm a big city kid I know that if I a always lock it the chances that someone is gonna steal it go way down. In small towns I could get away with not locking my bike but i still do it because if it got stolen I would be walking. Reason enough for me. I've only had one bike stolen (knock on wood) and it was locked in downtown chicago overnight with a cable lock.
That said i almost never bring my panniers in, but i take things of value out of them ( like tools) so someone else won't (learned my lesson that hard way about that one)