Bluefoxicy is talking the most truth. Be wary of where your calcium is coming from. High milk intake in some cultures is linked to brittle bones and higher fracture rates. People assume calcium is good for bones, but as Bluefoxicy indicated, it can actually leach calcium out of your bones rather than adding to it. Read more about this here:

http://www.hsph.harvard.edu/nutritio...ory/index.html

As to the OP, as long as the excess milk in your diet isn't effecting your cycling performance or digestion, it should be okay. Just an interesting note: David Zabriske just rode the TDF this year as mostly vegan (with some fish) and he put up some of his best stats this year as a vegan. He went to a chiropractor who decided he was very sensitive to milk and a lot of other foods. He drinks VEGA, some expensive protein drink designed by a famous triathlete. I drink cheap chocolate soy protein drinks with spirulina and it's great. Kinda like chocolate milk, but from a veggie source which has higher quality proteins and antioxidants.