Sounds like you have the right things in mind. Taking it further, I'd see how your mates feel about it. It's always nice if a club has ideas other than what it was started for. I think as long as you are realistic about individual and group expectations you'll establish a base to work from. Looking further down the line, you need to guard against factions developing, but that'll only be the case when it grows to a larger number. By that time, you'll have to replace the discussions of where you ride over a pint of beer, with a more formal organisation that tries to structure events.
For now, go for it as a social between mates but with an eye open for increased involvement. Adverts in shops (LBS might be a good place to go) may catch others who want to do the same but sort of wait for others to get it going. If it's with mates at the start you'll know the relative distance, riding styles, time duration that everyone is used to. If others join later, you'll need to plan a different set of criteria. I once joined a road club many years back. I kept cramping on my first ride with them and I soon fell behind. I got back to the meeting point last, and only my car was there. All others had gone. I didn't expect everyone to fall back to my pace, but I certainly expected the group leader to see what was going on ahead and behind. Needless to say, I never went back.