I thought you were at fault until I reread and noticed that the guy behind you wasn't associated with your ride in any way.

If you WERE on a group ride, I'd say you would have been at fault. I just did a line ride a few weeks ago, and our 5-man group was doing well, but passed by a fast trio. They unfortunately passed us right when a section of road hazard cone markers was lined up, and our front guy had obstructed view, so he couldn't signal to us, and had to swerve. I was on his wheel, and the moment he swerved out of the way at 21mph, I was running right into a cone. I barely made it, but the two guys behind me hit the next cones and went down hard. It's REALLY important for the leader of a line ride to signal and not swerve, because the guy behind you has no chance at survival if he's riding tightly on your wheel.

In your situation, I think the guy behind you was behind a rude dork. You never indicated that you knew he was there, and it's irresponsible to automatically latch onto solo riders and expect them to give you all the cues as if it suddenly became a group ride. If you're going to draft a solo rider that doesn't expect you there, it's fully in your responsibility that you have adequate visibility and response distance in case they swerve.

The only defense for the other guy is that he's probably used to riding mostly in groups, where doing the group signaling is natural and automatic. If you've been doing that awhile, you're sort of shocked when people don't do it automatically. This happens a bunch when I alternate riding between an advanced road bike racing group, then sometimes go on easy slow rides with a slow beginner-early intermediate tri group - I know most of the tri rookies don't know how to do the group signal thing, but it's still unnerving for the first 5 minutes with them.