I believe that I have the same combination of rotor (SM-RT78 from Nashbar) and brake (Avid BB7 road). I also had problems with weak, pulsating brakes. If you look closely, the brake track on the XT rotor is narrower than is typical. This means that your pads are probably hitting the arms, which causes the pulsing sound.

Using the XT rotor, the stopping power is about the same as the cantilevers that came on my Crosscheck-- not terrible but not as good as I'd like. The pulsing goes away and I have better stopping power when using an Avid Roundagon or a Clean Sweep G3. This solution works for both 6-bolt hubs and Centerlock hubs using a Shimano adapter (SM-RTAD10). A cheaper solution that doesn't work as well is to shim the brake mount out by one millimeter, so that less of the pad is hitting the arms.