I picked up cycling again 4 years back after a gap of about 38 years, partly for exercise.
First outing was 8 miles and I nearly died.
Now my training route is 21 miles 2 or 3 times a week.
That's just to point out that most people would be roughly where you are (I'm 90kg and 183).

But a point I'd like to make is that speed isn't everything, you could stay at a moderate speed and just go further or hillier.
I'm an old git and fully retired so I don't really feel the need to be fastest.