Just a FYI rotational mass is different from a static mass.

Adding (or removing) the same amount of weight on the tire/wheel VS anywhere else on the bike you will definitely feel the difference on the wheels/tires where it would be barely noticeable adding/removing on the bike as you experienced, and is not just in your head.