He could have been PO'd. Hard to tell without being there. Unfortunately there are a lot of riders that drop ambiguous comments like that. Makes it hard to know whether or not you're doing the right thing if you're new to the game.
Generally speaking if he pulled you guys for a while, and then started to falter, and you go to the front to pull....don't drop him. Hold back with him allowing him to ride in your draft to help him recover a little bit.
If he's totally shot he'll probably tell you to just go on. Most people don't like holding others back.
In this situation it is hard to tell because you mention you weren't riding with him until you would catch him on the hills.
...ahhh....it's finally spring....group ride etiquette threads begin!