Some time in the mid 90s I spoke with someone who had worked on this kind of problem (and others) on the London Underground. There had been a desire to know platform occupancy, and that this could be achieved by using a NN. The story as relayed to me was that the NN was never much more reliable that simply measuring the brightness of the platform — or rather the reflection of lights from the platform surface — which could be fairly easily calibrated
I took it less as a cautionary tale about NN and more about looking for simple solutions
My AI lecturer at University told both the tank story and a similar London Underground story. In his version of the latter, the NN was measuring platform occupancy but sometimes produced anomalous results. After investigation it was determined that the NN was actually counting the number of bins that were visible on the wall (pre-IRA, I guess). On average, the more people the fewer bins are visible.
I took it less as a cautionary tale about NN and more about looking for simple solutions