If you are serious about riding and you aren't going to be buying new frames every year, I'd pick the better frame - I think the Roubaix is going to be better in this area. Later on you can upgrade components, but if you have never ridden on high end components you aren't going to notice a huge difference between 105 and Rival. Make no mistake though, the Rival is superior.

That said, I plan on buying a Roubaix when I can. I ride like you do - long solo rides with headphones on and then a bunch of charity rides. I do a lot of 60-100 mile rides lately and I'd like a Roubaix. I have some neck problems (bulging disc) so I'd much prefer being able to sit up a bit.