Just ride without him for a 30+ mile ride one morning per week (get up early and go). Within two months you will be leading the way.

Or ride one steep-***** hill for 15 miles, once a week, without him.

Either way, if you train harder and more frequently than him, you'll pass him.

Do check your tires though. If you've got 650x28c treaded tires, and he has 700x23c slicks, he's at an advantage. Not because of the 650 vs 700, but because of the tread differences, and width/pressure.