Ok, first of all if you are non-competitive than the time savings an aero rim gives you means nothing.
Fortunately, you can have light weight and aerodynamics with Zipps or whatever. The downside is that they are harder to control in crosswinds but if you are going for one race wheelset, I would go for 303's, 404's, reynolds carbon, etc.
Overall wheel performance is much more important than the aero benefit. This is the reason Ksyriums or custom laced "box" section rims like a mavic reflex is popular in the peloton. Especially in road races.
The number one factor is the rider preference in wheels. To say someone is stupid or wasting energy for riding non-aero wheels is really ignorant. Take a look at any recent road race and see how many of the top riders are on carbon aero wheels and how many are on standard wheels.
In theory, aero carbons are great and I love them. I also love those campagnolo hyperions. INcredible wheels! They are carbon rim, but not aero. They are beautiful and fast.