On a tube, the farther the material is from the center, the more it contributes to stiffness. So, it's possible to hollow out quite a bit of mass from the center and save quite a lot of weight, while losing relatively little stiffness. However, you still lose a little stiffness; it's not like the air in the tube is helping hold the aluminum around it in place!

When tubular cranks are stiff, it's because the arms are wide; we're getting to where some of the premium cranksets look absolutely silly to a lot of people.

You see the same thing in frame design. Increasing the thickness of the tube's walls increases the bike's stiffness, it doesn't decrease it. However, it's a fairly small effect compared with increasing the tube's diameter. Hence, steel frames with skinny tubing are often pretty flexy, even when they weigh more than aluminum frames, even though steel is similarly stiff-per-weight as aluminum.