Lots of group rides start out with a warm up and there is lots of talking. Basically, if I'm not working hard, I'm talking to the guy next to me. As the pace picks up, the talking decreases. As you have experienced, it is quite annoying to be in a massive suffer-fest and the riding partner is still chatting. I still make this mistake when I ride with my girlfriend. Often times I'll do intervals or hill repeats while riding with her. She is happy that she is not "holding me back", I'm happy that I get to suffer, and she gets some quiet time during the hardest part of the ride.

"I prefer to suffer in silence." is probably a good response and the bf should understand.