There are some bridges out on the Kissing Bridge Trail, so some on-road diversions (or stream fordings) are required. Also, the maintenance gets much worse toward Millbank.
I agree that the TCT can be difficult to follow, and is somewhat disappointing in the sense that so much of it is on-road, especially through cities.