To correctly answer the question we need to know your body fat percentage, and racing goals.
If your body fat is below say 8% then I'd say don't go lower.
If you are anything but a climber or MTB racer I'd say no. For crits, RRs, and TTs you'll be fine at 180.