Not generally a problem, though if the tubes have to be folded up and squeezed to fit, there's at least a theoretical chance of rubbing or pinching a hole in them. FWIW, I buy only two sizes of tube: 26x1.5 for our mountain bikes and 700x25 for the road bikes. They go into everything from 26x1.4 to 2.3, and 700x23 to 41. No problems ever.
What's "wobbly," though? If you can feel the tires squirming around on the rims, check your pressures. Don't run the skinny road tires at the same low pressures you ran the big knobbies.