That same feeling was creeping up on me as I was getting ready to complete my first century. Along the ride, I met someone who rode along with me the rest of the way. He had a jersey that said Shenandoah 1200k. I asked him about it, and he explained the world of randonneuring.
It's a natural next step from the century. If you can do 100 miles, you can do 128 (or 200k), and then you're on your way up the ladder. Check out
www.RUSA.org for more info if you are interested.
Endurance racing is another way to go. There are many options for 24 hour races. You don't need TDF speed for these events. It's a combination of a lot of skills, nutrition and such, and a stubborn attitude that can make you competitive.
www.ultracycling.com has links to many endurance events.
I find it helpful to have some long term cycling goals. But, there's nothing wrong with just enjoying it. Presumably, you are having fun, getting healthy, and saving some money. Anyway, good luck whatever you do.