I carry a wallet for day to day purchases, with cash and an ATM card inside. For my passport I have it in a waterproof case (along with my credit cards and HDD), placed inside of waterproof bag, which stay in my rear panniers. I am far more likely to lose my passport than my pannier being stole i.e. I am extremely forgetful.

You can't waterproof you passport enough. Bad experience....