OP, you're mixing apples with oranges.

First, going from your 20-40 miles to 100 miles in a weekend is a huge leap, especially since you started riding seriously in January. Ride your 20-40 miles on your road bike and see what happens.

Second, you've mentioned that you could only keep up with the roadies for a dozen miles on your fixed gear bike. Go for another ride with the group on your road bike, and see what happens.

Then report back with results.

As for using the same gear inches, as recommended by a couple of knowledgeable people here, that'll be interesting. However, you got your bike to go faster. If you want to go faster, you do that by pushing bigger gears.

Of course, you'll be able to pedal on the downhills, too, at least the gentle ones, a lot faster on a road bike than on a fixie. On a fixie, as you probably have discovered (unless you live in Florida), you spin out fairly quickly with a 71 inch gear on the wimpiest of downhills. That won't happen when you're turning with something like 50/11.