Quick release water bottle cage:
Bicycle Bike Cycling Universal Water Bottle Cage Rack Holder Mount Quick Release | eBay
You can clamp it on seat post or steering post or handlebar, and remove it when it's in the way (such as when bike is folded) just by flipping open the quick release lever. The direction of the cage can be rotated so it mounts to a horizontal tube or vertical tube.