There are some people who are relatively fast into their 70s. Yesterday I rode with our club's 'B' group and there was a 73 year old guy who stayed with the group the whole 65 miles and did very well in the hills.
I do think riding with faster people will help your speed, as long as they don't leave you in 1/4 mile. Are you aero? Getting low will save energy.