I think more accurate to say "You can't decide not to build a bridge, based on the fact that not enough people currently swim across"

If the need to get across is so great that some people are currently swimming, that sounds like a pretty good justification for a bridge.