It's not that mysterious.

• There aren't many long traffic-free loops in the general area.
• Prospect Park is just as crowded.
• If you're doing a short training loop, it's the most efficient place to do it.
• IIRC a couple of racing teams train (and race) in the park.
• Many Manhattan cyclists are highly reluctant to head over to, say, Palisades Park on a regular basis.
• Some (if not most) cyclists feel, when they are on the bike at least, that the park is really "theirs" and that pedestrians, children, skaters, joggers and dogs (especially dogs, which are allowed to go off-leash in parts of the park) are merely obstacles that have no real business in the park.
• Last but not least, I'd guess that the fast cyclists figure they can handle it, that they're skilled enough to dodge the "obstacles," and don't view it as particularly unsafe.