Actually, in Toronto it's an offense for pedestrians to "fail to yield to traffic" under the Ontario Traffic Act, Section 144(8) (as per
this article). Technically, they don't have right of way. Right of way only applies here in cross-walks, not mid-street level.
A little further to this is Section 144(7):
Yielding to pedestrians
(7) When under this section a driver is permitted to proceed, the driver shall yield the right of way to pedestrians lawfully within a crosswalk