It's funny, but even though I used fanny packs a lot cross country skiing, and all the time during my photography career doing press type jobs, I actually don't like the way they feel.
This does depend on how much weight is in them, and with more weight and wearing them for significant hours at a time, I always tended to shift then around a lot. Moving it to the side put less strain on my back, especially with long photo jobs where I had usually three lenses in it and adjoining lens pouches, plus a heavy camera+flash around my neck or on shoulder.

So because of these experiences, I've never tried them riding, but strongly suspect I wouldn't like the sweaty feeling and weight and most likely the slow loosening of the strap (pretty to common with xc skiing, and also a sweaty lower back)