Depends on the tri saddle. If it's just one that has more padding in the nose, the only downside is weight and looks.

The funky ones, like the Adamo, in my experience do have a handling downside. I put an Adamo on our tandem when it's set up in time trial mode. It helps with the aerobar position, but it also feels like I'm sitting on top of a board, not wrapped around the saddle.

A lot of steering comes from the hips. If the saddle weakens your connection with the bike, it hurts the handling. The difference is not huge, and I've adjusted to it, but its noticeable.

Bottom line though is use what fits you and is comfortable for you.