3 days a week should be plenty IMO. Of course it completely depends on what you do with those days. How much time do you have available to ride? I started with a similar routine a few years ago - I'd have lunchtimes or early mornings to do a hill climb and ride for an hour to an hour and a half and then a few hours one morning over the weekend (kids, chores etc. tended to define that for me). What I found helpful was to use the shorter weekday time for more intense workouts, and then build distance on the weekends.

It's totally doable.