A fictional variant of this story also shows up in Peter Watts' novel Starfish.
In that version, the neural net is trained to manage the ventilation system of an underground train station, based on data about passenger movements. Unfortunately, it ends up simply paying attention to the display of a clock that happens to be visible through one of its cameras. And when the clock breaks, a bunch of people asphyxiate. A bit implausible, but memorable.
Some time in the mid 90s I spoke with someone who had worked on this kind of problem (and others) on the London Underground. There had been a desire to know platform occupancy, and that this could be achieved by using a NN. The story as relayed to me was that the NN was never much more reliable that simply measuring the brightness of the platform — or rather the reflection of lights from the platform surface — which could be fairly easily calibrated
I took it less as a cautionary tale about NN and more about looking for simple solutions
My AI lecturer at University told both the tank story and a similar London Underground story. In his version of the latter, the NN was measuring platform occupancy but sometimes produced anomalous results. After investigation it was determined that the NN was actually counting the number of bins that were visible on the wall (pre-IRA, I guess). On average, the more people the fewer bins are visible.
Sure, but the average person, on seeing just a "it's good!" review, might be surprised by the hundreds of pages describing the viewpoint character being raped.
In that version, the neural net is trained to manage the ventilation system of an underground train station, based on data about passenger movements. Unfortunately, it ends up simply paying attention to the display of a clock that happens to be visible through one of its cameras. And when the clock breaks, a bunch of people asphyxiate. A bit implausible, but memorable.