I don't know how cyclists ride in Toronto in general, but if your boyfriend exhibits the norm and you deviate from it - no matter the propriety or legality of the norm - that deviation may be the "problem." It may well be that the drivers are expecting one thing, for right or for wrong, and you aren't doing it, thus creating in the drivers' heads the impression that you're an annoyance or a hazard to be avoided.
I think that the best people to tell you why what's happening is happening would be others from the Toronto area.