I think you're right. I've been experimenting the same way and see it the same as you. I doubt it's the relative difference in speeds. If a car is going 40 they probably can't tell much difference between you at 15 and you at 20. It's the bicycle rider that feels like there's a big difference.
It could be like you say, they see you trying hard so they're more understanding. Or that you seem less predictable. I don't know. I tend to think the later. I guess I think people are pretty selfish about their "mission" in a car and don't give you much consideration out of sympathy. It's more to avoid an accident.
So I wonder if I/you could go slowly but look like you're trying hard??