For me, it's the challenge.

I'm currently working my way through two series of long distance rides: The R-12 Award is earned by riding a 200km (or longer) randonneuring event for 12 consecutive months. This can be a challenge for many reasons... The logistics of dedicating a weekend every month when you have job and family obligations, or (as is the case usually) the weather. Some areas it gets gnarly hot in the summer, for me, it's the bitter cold in the winter. Finding a route without snow on it is often a difficulty, and I've ridden many a 200k in the snow, starting before the sun is up and finishing after it's long been dark again.
The other one I'm working on is the Super Randonneur, which is a 200km, 300km, 400km and 600km event all in one year.
For me, riding isn't just about getting from one place to another physically; it's about reaching a new mental level. You learn a lot about yourself by spending 12 hours alone riding in snow and freezing cold rain, or pushing yourself for 24 (or more) straight hours. Few people understand what those keychain sized trinkets of accomplishment we earn really mean to us.