I refill most often at public spigots. Make sure to look around the supports of any picnic shelters you see - pretty often I see parks with no water fountain, but there will be a spigot down by the ground somewhere for picnic shelter renters to use. Beaches will often have water to rinse off your feet with. Park maintenance buildings may have have a spigot on an outside wall. Keep an eye out for public or community gardens, too, but these are the only places I've seen where the spigot has on several occasions been locked. Even if there is no water in a park, there may be people, and if they start asking about your bike or trip, mention you're looking for water and see what happens.

If you run across a town with a big national grocery store, those are great places to walk in and refill your water in the bathroom without anyone noticing or caring. Small groceries or gas station are my last resort because if I do run across sometime who refuses me water, I'll run it over in my head all day(Why would they do that? Is there something wrong with me? do I look too scruffy? Homeless? Do people come in here and demand free water rudely?). When I do, I'll usually buy a snickers bar or something, then ask if I can refill my water on the way out. Depends on rather the cashier seems approachable or not.

Also, anytime you fill up your bottles, try to resist taking a big gulp of it. Smell it and taste a little first. Non-potable water spigots should be marked, but they're not always. Even if they are marked as safe, you never really know - I just read a journal in which a cyclist took a big swag of bleach water at a state park, because a maintenance worker had just flushed out the pipes and left no warning of it.

All in all, Though, yeah - I'd agree it does just work out. I've never ran out of water. One time I was on my last bottle of hot sun-baked water and a bit worried about where I'd get the next. I came across a landing on a river, and stopped to cool off and rest. I didn't have a filter, and could not drink the water, but while I was soaking in it, a family came by in canoes and offered me some ice water from their cooler