Shimano jockey wheels are made with bushings with a slight amount of side to side motion, with the thought that it helps with the shifting to find the gears better.

Ultegra and Dura Ace jockey wheels should have ceramic bushings, but 105 should have steel bushings. They should still be rather long wearing.

I have chosen aluminum ball bearing jockey wheels on one bike because I thought the older derailleur had too much flex throughout to shift well.

If you like how you're configured your bike, then go ahead and ride as is. Then you could try upgrading to some Ultegra wheels, or try the ball bearing wheels, whichever you think is best.