If he was looking over his shoulder then he didn't have mirrors and was simply doing what you yourself said that you failed to do. Don't worry about it. I look over my shoulder all the time to check the traffic siituation.

Regards, Anthony