Incremental training, preferably on a dedicated training plan is the best way to go. Probably virtually nobody here on BF does it, but for people with a running background (me), the effects are clear and obvious.

Training "casually" usually gets you up to where your hard comfort zone is. Pushing yourself out of that area takes repeated hard work, with a lot of time spent doing rides in intensity zones that you don't think you can do and don't want to do. But with the right plan, you'll have the knowledge that you've got the preparation to hack it, and you'll be able to do it.