Middlebury, which is 45-50 minutes south of Burlington, has culture (the college), plenty of stuff to do that is not cycling-related, and the option for flat rides in the Champlain Valley and hilly rides if you go east into the Green Mountains. And the Black Sheep Bistro in Vergennes, 15 minutes' drive north of Middlebury, has the best french fries in the known universe. Even if you're based in Burlington, a visit to Vergennes and Middlebury would be worthwhile.