First does Philly have any laws that require you to remain in the bike lane if one is present? If not then it is or should be legal for you to take the lane, particularly if you are traveling faster then the other traffic in the bike lane. Also given the surface hazards that you noted that are usually in the bike lane that should also strengthen your position for taking the lane instead of riding in the bike lane.

That up to this point no other drivers had had a problem with your actions I would say that it wasn't a problem, except to this one driver as I presume that you travel the same route at about the same time just about everyday and the motorists are used to seeing you there.

Your concerns are legitimate, and as has been said before we are the one's who get to decide if the conditions to the right or in the bike lane are safe for us. You could/should have done as you did pulling over into the bike lane and then once the SUV passed moved back into the travel lane.

When a friend of mine and me ride together he'll ride in the bike lane, and I'll ride in the travel lane so that we can talk. And so far we have had no problems with motorists with this arrangement. As checking my mirrors, if I see a car coming up behind us I'll either speed up and pull ahead of my friend or slow down and drop in behind him. And then once the car passes move back to where I had been.