You should definitely go. Even if you just stay in the area around town and don't venture too far, it is worth the trip.
Block Island is a great place to go- though riding is basically the 17 mile loop that is on most of the maps (southern loop with the in and out north) but there are a lots of gravel roads as well.
I used to take my daughters and wife there when they were kids and they are not cyclists either but they did OK. It isn't flat (as one might imagine by being an island) but the hills aren't really bad or long. People walk them.
The riding there is absolutely beautiful- in fact there are so many scenic stops and vistas that they would probably interfere with what some would consider a good workout. There are so many places to park and view and/or take walks. (bring a lock)
Also what I did was leave the family to shop and/or stay on the beach and I would go riding and get some more miles in.
PS The thing I love best is riding right along the shore or at the south end riding above the water with the sea going on forever-you can't get that at many other places around here.