I don't think it's particularly rude, but it's definitely unsafe.
If you're within drafting distance, you have very little space to react if they decide to stop. Someone who is riding alone or side-by-side isn't likely to signal his slow downs and a lot less likely to signal an emergency stop. This means that you get to run into them if that happens.
Otherwise, I don't really think it's "rude." Sure, it may feel uncomfortable to feel like someone is following you, but most bicyclists know what drafting is. You being behind them would only affect their aerodynamics slightly, and more likely positively, as it would reduce the effect of the ow-pressure and high turbulence zone right behind them (but this effect is pretty much negligible). Basically, no, you're not causing them extra work.