I can't say whether XC racing attendance around here is up or down (I have only done two races... both of which were well attended), but cross seems to be pretty hot right now. I think one reason it may increase is that it draws both the roadies and the off-road racers (it's in the off-season for both).
I'm not sure how well you can compare the attendance of road racing to off-road, though, because you need to account for population differences. I was talking to some roadies yesterday at a night road ride who probably have never ridden dirt before. In general, if you find someone who rides a bike, especially in races, they would be a road biker.