Usually the first time I'm in CP every year the Harlem hill is a killer, but by the end of the season, it isn't so hard. The rollers after it seem like nothing once you're in shape, but you wonder what happened to your body when you're not. As for the downhill, because it is basically immediately before the uphill you don't feel like you get any benefit from it.
Prospect Park in Brooklyn can be even more tedious as the loop is shorter, but it is much less crowded, even on weekends. The hill is less steep but longer. At least that's how it seems. I've done PP hill at 20+, even my 12 year old daughter does it at 18, although on bad days I drag up at 14.