Hermes is 100% correct in his assesment of proritizing stages for developing faster average speed. I would add to this the importance of comradery in riding with a group. Yes, conforming into looking like a cyclist is important and improving ones eating habits along with pushing harder on rides is absolutely vital to improving ones speed.

I think it is just as important to remember to keep it fun and safe. You need to find a group that has a "no rider left behind policy" so that at least one member stays back with you until your avg mph speed improves. This could take weeks, months , or years depending on how well you adapt to the factors Hermes mentioned above...