I don't think you're doing to much. If you can do it without pain than why not? I started off exactly as you are and by the end of first summer I did a century. Now the only time it hurts is when doing hard training rides. You're doing great start adding on the miles but remember to take a day off and mix in some light easy recovery rides.