Had 46mm on the Fuji, great all around size, imo. Zero handling issues, and held speed well. To be honest tho, I didn't notice any difference in speed between the 46mm carbons, and the 32mm alloys. Lil more comfort with the 46s because they were wider and I had wider tires on them. And they looked better, I'm not afraid to admit that aesthetics matter.
Building a Tarmac, will go with 30-32mm wheels to save some weight, plus aesthetics (I don't like the look of deep wheels on frames with smaller tubes, they look fantastic on an aero bike tho).
Not building an S-Works (I wish!!), just using this pic for visual reference. IMO, the 32mm wheels look much better on this frame.