+1 for what jppe says.
What you are doing riding-wise is fine and is probably better than most of the people you see on your rides, but I also detect this desire to get faster/fitter. This can be done, but it is not the most enjoyable endeavor! The key to getting faster is interval training and it is not a pleasant experience. It is one of those things in life where if it doesn't kill you (and it will feel like it is killing you!), then it will make you stronger.