Am I correct in assuming that these are the bars that have a cross brace that is keeping you from just removing the grips and sliding them off?

If so, I would try removing the bolt completely from the stem and then trying to slide the cross member through the open part of the stem... and perhaps prying the stem open with a large screwdriver to expand the opening.