No problem on 23's. I've ridden it many times on a fixie with 700x23 wheels.
I would suggest making a stop in Chemainus, it's probably the funkiest of the towns along the route. They've got lots of murals painted on the buildings, plus nice restaurants, even a community theatre. Cowichan Bay is quite scenic. I would also recommend taking the ferry from Crofton and doing a side trip to Saltspring Island. Book something in Ganges, another funky town. Be aware that Saltspring is a pretty hilly island, though. But the side roads are pretty quiet.
Luis