I think it's mainly because drivers don't trust a cyclist do obey the law and not do something stupid. When I'm driving my car, quite frequently I see cyclists do extremely stupid things like cutting across 3 lanes of traffic without even indicating making cars slam their brakes on to avoid running them over.
I know that none of us here would do something like that, but the car drivers just don't trust cyclists. They see just one do something illegal and their brain automatically thinks that all cyclists are the same. It's like that with most things, you don't notice something unless you're specifically looking for it. Someone in a car won't pay any attention to a cyclist riding along obeying the law, but if they see one run a red light, or cut someone off, then they'll start noticing and think how irresponsible we all are.