I don't see why there's anything wrong with it, although you should be careful that they don't stop suddenly. If he's irritated, he could slow down and let you pass.

Once, I was riding my fixie, and two other guys (I think also on fixies) were following me. After going for a while, I decided to rest and think about what route to take home, since it was a good deal later than the time I'd told my wife I'd be home. I had a front brake, and I'm not sure whether they did or not, because I think they might have come close to hitting me. If I'd thought of it, I would have let them know I was stopping.
Then there was a short downhill branch with a sign saying bikes one way, and a longer branch. I decided to go the wrong way (uphill) on the short branch to save time. They caught up with me later.