If you ever stopped drinking milk for a long period you'll stop producing the enzyme required to digest the lactose. That will give you trouble. Personally i think milk is an ideal food. Maybe some objections are based on it's source? I know, almond milk or soy milk aren't without health risks either so do your research before taking those up also.