Geeze, you sound like me, same age an' same average speed!
Sounds like you are doing good to me. Try to mix things up to keep it fresh. I climbe hills on relatively short rides during the week and on the weekends try to go out for longer road rides.
My average speed has slowly increased over time; very slowly because of the hills I must climb to get back home to my house. I am noticing better speed on relatively flat routes and am making better through speed up the hills. By through speed I mean I don't have to stop at all or as much on big hills as I used to.
These are my indicators of improvement. I want to ride a century (miles) in a day eventually, but that will have to wait till I can find a saddle that I am comfortable in when riding over 1-1/2 hours.
I also want to go as fast as my husband does when he's "not riding very hard" and doing 7 miles an hour up the hill to our house. I average 4-1/2 to 5 miles an hour up the same hill. Some day!
Keep at it and don't burn yourself out. You should be able to smile, laugh and enjoy yourself when you go out! Have fun! Don't ride serious ALL the time.