HoustonB, first you say you're riding as far right as possible, then you say you're taking the lane. Which is it? Those are very different situations.
But even if you're taking the lane in a centerish position, you must yield to someone who may legitimately be passing you on your left before you veer from your line of travel. Of course, you can veer a foot or two, because anyone passing you should be doing so with a safe passing distance, generally considered to be about about 3 feet. But if you're going to move laterally more than a foot or two, simply signalling does not automatically give you the right to do so, nor does it require anyone behind you to yield to you. They still have the right to pass you on the left, if it's safe and reasonable to do so, and you're required to yield to them (you're not allowed to turn in front of someone who is passing you just because you signalled before you turned).