I recently bout a new wheel, 8-speed cassette, and chain. The cassette is a HG-50 (RSX) 13-26 and a HG-91 chain.
When I ride on my rollers, I notice a horrible vibration and grinding sound when in the 15 tooth and physically smaller gears (13 & 14). It is silky smooth in all of the other physically bigger gears (17 - 26). It is the same regardless if the front is on the 39 or 52.
As far as I can tell, this is due to the chain rubbing on the physically larger gear of the cassette, but I'm not 100% certain this is the case. This cassette has never been taken apart.
Also, when I ride on the road, I don't notice it at all. So, I suppose it's not as bad as I made it out to be. I'm sure it's still there, but all of the other vibrations and such mask it.
Have you tried the cycle in a bicycle stand and watch the rollers and cassette whilst pedaling. Bummer if you haven't got one. Easiest way of finding that kind of problem unless it's chain pressure issue.
Most likely when the bike is on the stand and the rear wheel is spinning in air, it's hitting some resonance frequency that causes the vibration you feel. On the road with weight on the wheel, the dynamics are different so the resonance does not occur. Since you state that you do not feel the vibration while riding, I would not worry about it.