A few points in this ride description stick out to me. One, fixed gear pace vs. geared, 2 strangers. Two, you weren't sure if he even knew you were there. Three- he did, and he ditched you when he got the opportunity.
To me, there's an inherent mismatch in pacing with fixed vs. geared. This makes it an awkward dance at best, let alone with someone you don't know. Hanging on someone's wheel in that situation is risky, and I wouldn't be comfortable with it.
I don't mean to single out the OP- roadie's have a deserved reputation for d-bag attitudes. The guy on the geared bike could have been nicer, or said something sooner. Most cyclists are pretty friendly, but every few rides I run into people I wonder what their deal is.