Do it.
Galloping Goose Trail goes from Victoria to Sooke. I rode from Swartz Bay ferry past sooke on a tandem with wife+2 kids. It was awesome. Riding the hilly/narrow shoulder on the west coast road past sooke is less fun.
Maps and info here:
Galloping Goose trail Vancouver Island