Yeah, when a metal frame flexes you lose very very little of the energy as there is no damping. Does the energy get back into the pedal stroke? Maybe, maybe not... but it really does not seem to slow anyone don too much. Back in the '70s and '80s a company called Vitus made narrow-tube aluminum bikes and they flexed like crazy, but were still very fast while climbing - even though climbing is often at the low-rpm situation that result in more frame flex.
All that being said, I'll bet a whole lot of the flex you were noticing was actually in the wheel and not the frame... spoked wheels flex much much more than any frame when subjected to a side load.