Yes, I remove my panniers every night at home. I'm using a front rack, which makes packing the bags tricky while on the bike. I don't lock my bike up outside, and if I did, I would certainly bring the bags with me. The removal method involves just lifting with the handle, so they come off instantly. If you align everything properly, and if there is enough weight in the bag, the method of installing it is by dropping it onto the rack, so it's about as quick as removal.

I don't know much about racks. They have new rear racks which hold panniers very far back, and that prevents your heels from scraping the bags, and that's a nice thing. This is a problem only on some bikes and with some bags and some feet, so you may be best off rack shopping at your local bike shop.