As a first approximation, it doesn't matter how fast you ride, a slow rider will use calories at a low rate for a long time, a fast rider will use calories at a high rate for a short time. Same distance, same calories.
On closer examination, the fast rider is fighting against much higher drag forces. Aerodynamic drag increases with the square of velocity. Double the speed, quadruple the aerodynamic drag. The fast rider would use more calories to cover the same distance.