I would say it is one thing to tip over in a kayak and get dunked, and another to allow swimming. You don't want to tip over, and the kayaking at the piers is meant for you to stay upright. I've seen kayaking in Newtown Creek, can you imagine diving into that muck? I wouldn't kayak in it either as it is so loaded with toxic waste.
There have been articles too of people moving into spaces that are directly on top of the toxic waste sites in industrial Brooklyn. There was a reason nobody lived there and these neighborhoods were deserted.