Aero trumps weight for time-trialing like triathlons. Not so for road races or criteriums where you have to repeatedly accellerate the wheels. Given that this is a tri forum, you'd presumably be better off with a slightly heavier, but more aero wheelset.
But the point about your weight and being able to keep the wheels stable is valid as well. Aero wheels can be difficult in a cross-wind.
How much time aero wheels will actually save, and whether it matters are different questions altogether.