You're right, and the physics is not very complicated. Take a bike with total mass M, and two wheels, each with mass m and radius r. The kinetic energy (translational + rotational) of the bike moving at velocity v is

e kinetic energy of the bike at any given speed is completely independent of the wheel diameter, as long as the larger/smaller wheels have the same mass. However (this is interesting in general) note that adding a gram to the rim has twice as much effect (adds to both "m" and "M") as adding a gram to the frame (adds to just "M").

http://xkcd.com/54/