Sorry to be the downer here, but if these roadie guys are any good: they will smoke you, especially on the downhill.
THIS
im in good shape and a proficient cyclist, and i still find group rides challenging with my geared bikes. can't imagine doing it on a fixed or SS
any worthwhile group has some pretty good riders in the bunch. i dont see you keeping up with their pace.