"PS, most of them ride 3 nights a week while Im on rollers or on the road 1 night a week."
How can you do any endurance sport well if you only do it one day a week. That make no sense. If you aren't interested enough in cycling to ride most days, you don't belong in the fast group. Cycling obviously isn't your thing (yet anyway).