I looked at side car and chose not to get it because:
1. overall width + where I ride. I cycle on roads that are narrow and winding that lack a bike lane or even reasonable shoulders. I ride in traffic and was concerned that the sidecar would make me too obtrusive.
2. drag. Might not be an issue for you ex-racers, but I feel like I'm hauling around a 50 pound parachute everywhere I go (I pull a Chariot CX trailer behind me these days). I know that I'm grasping at straws, but I just couldn't see how putting that parachute next to me was going to improve my cycling speed or reduce the amount of effort I am putting out.
3. quick switches. I cycle with trailer in tow on my single bike during the week. On weekends, we easily and quickly switch from single to tandem. I don't remember the side car having that flexibility.