What RainmanP said

As for why not to ride in them all the time, when there is a chance that something might jump out in front of you, its good to be closer to the brakes. On clip on aerobars you won't be able to change gears. Road races won't let you use them. Other group riders may not like them because you can't point out pot holes easily with them (I found this out when drafting someone closely and they were in their aerobars, they avoided the pothole and I got a flat). You might also impale someone on the ends of them. Climbing hills is easier when not in aerobars, but then you'd gain little aero advantage going up hills anyway due to the reduced speed. Extreme turning is also not recommended under their influence.

Personally I use them only when out in the country and generally only when on my own and when speed justifies it.