Statues, monuments, etc. are one thing. The confederate flag has a special significance in our modern era. The reason for this is that when the Dixiecrats split off from the Democratic party around 1948 because they opposed desegregation, the confederate flag was resuscitated as a symbol of resistance to desegregation, anti-lynching, civil rights, etc. Virtually everyone waving that flag today are doing so because of that political legacy, and so it is clearly, unambiguously, and exclusively a symbol of racial oppression.
God help us if people stymied by activism against the confederate flag decide to pick another symbol (such as General Lee) to use as a substitute.