Actually a large turnout on my club's breakfast rides is 12 cyclists.
Usually we'll split into slow group and fast group so yes it isn't unusual to have < 6 people in a group.
We'll get more people showing up for charity rides (Tour de Cure, Hotter n Hell) or special events (advocacy, mayor's rides, parades).
Many of our folks also belong to other area clubs and pick and choose between offered rides.

Most of our club are middle-aged engineers and management for a Fortune 500 company and have plenty of personal disposable income.
Certainly we would cut some slack for those who would find it a hardship.