I live on the water, as obviously do all my neighbors. We moved here when my youngest was 2.5. Her sister was 7 and already swam well. We fenced across the back yard at the dock edge, just a 4 foot picket, with a hard to reach clip on the gate latch. The sides were already fenced with 4' chain link. Never had a hint of an issue. Once the younger one hit about 5 and swam well, the fence came down. None of the neighbors has lost a child. Dogs, oddly enough, are another story. They fall in or jump in to swim and can't climb ladders, and eventually run out of energy. You have to watch it with dogs. I know of at least two nearby that drowned.
Now, every child is different. There are those that try to escape from every prison no matter what. Then you not only have to worry about your backyard, but everyone else's nearby, and the end of the street where there is no fence at all. I recall once watching a neighbor's child from the next street, in January, climbing around a fence from one back yard to another on the opposite side of my canal. So she was hanging over the water, which had a thin layer of ice. Not good, she wouldn't have lasted long in there. We yelled at her to stop and told the parents of course, who were horrified. She was a bit of a wild one. But mine were never like that, at least not past that terrible two thing.
It comes down to the children you have and how close tabs you can keep on them. When they are small and they go outside, you have to stay vigilant. But to keep it in perspective, escapee type children can be hit by cars, or kidnapped by oddballs, when nowhere near the water. So it's maybe not such a huge risk in relative terms. And, they don't stay small forever. Key is, be aware of and manage the risk, and as Capt Art said, push the swimming early.