This may not be a satisfying answer, but the fat that accumulates under the skin around the waist is in cells that do not release fat readily in response to metabolic signaling about energy deficits. This makes that fat mass slow to go away with exercise and caloric restriction. Men and women have different subcutaneous fat distributions. Men tend to accumulate it in a relatively narrow area around the midriff. Women tend to distribute it "better."

As many people have written above, losing subcutaneous fat takes a relatively large and long-term negative energy balance.