I've done that, too.
My wife and I had the same reaction: surprised that it hadn't happened before.
Fortunately, I have my own office and access through a rear freight elevator. I worked with the door closed for a couple hours then made a beeline home and back, just 11 miles RT.
No one ever knew unless I told the story on myself, which I did frequently.