I'm in a similar boat, I'm used to doing 20-25 mile trips, 40 mile commute to work 2x a week (20 morning, 20 night) and recently rode on a few group rides at 16-18mph pace for 35-45 miles. I'm a clydesdale, and can stand to lose about 25 pounds.

Two things... one, riding in a group will motivate you to ride faster, period. You WILL push yourself more, and achieve more than you thought possible, if only because of the competitiveness. Two, if you ride 2-3x a week, you'll develop that speed and endurance in a matter of weeks. Go for it dude!