We have been doing it with our 11, 9, 6 and 4 year old since they were 3, 1, 0 and 0. Admittedly the only time we do the whole thing (~52 miles after the ride back from the ferry at the end) is when it is just me taking them. When the wife goes along, the pace slows and there is less drive to do the whole course. Not a bad thing though as it gives us time to enjoy the city more. If you take the bail out option of the Brooklyn Bridge, it is about 25 miles total for us. We finish that up somewhere between 12p and 2p IIRC. Doing the whole thing takes us to 4-6p depending on crowds and ferries.
I suspect we will bail this year unless I crack the whip some. My 9 year old is probably not up to it, but we have rest seats we can swap them to if needed (TAB, and trailer).
I assume you have tix? If not you'll be looking at the aftermarket to get them. Don't pay the double cost hawkers. As you get closer you'll be able to get them close to or even below cost.