According to this link, the kayaking from the Downtown Boathouse is only between Manhattan piers, not across the river:

Free Kayaking ? The Downtown Boathouse

I have seen kayaking in the Hudson River at the north end of Battery Park City and in Hoboken, and on the East River at Powell's Cove in Queens. All of these kayakers stay close to shore; none go across the river.

But if there is shore-to-shore kayaking somewhere, I'd like to know the details about it.

A side note: one thing that has always baffled me is the act of kayaking in waters in which swimming is not allowed. A kayak must occasionally tip over. So, if kayaking is allowed, then surely the water is OK for swimming, right?