It reminds me of a similar story about using deep learning on medical imagery for diagnosis. In one study, it supposedly learned the to tell which device was used to take the images. Apparently that was somewhat correlated with the results as more serious / heavily suspected cases were sent to a specific hospital in the region.
Similarly, I recall a talk by Daphne Koller where she talks about a heart attack predictor using X-rays as input. They found that stents were the biggest factor for positive prediction. Which is useless because the doctor would know if the patient had a stent.
A similar story I heard from colleagues in the field was about a CNN trained on images of tumors that ended up learning to identify images containing rulers that were placed there by a human for reference.
>Clever Hans (German: der Kluge Hans; c. 1895 – c. 1916) was a horse that was claimed to have performed arithmetic and other intellectual tasks. After a formal investigation in 1907, psychologist Oskar Pfungst demonstrated that the horse was not actually performing these mental tasks, but was watching the reactions of his trainer. He discovered this artifact in the research methodology, wherein the horse was responding directly to involuntary cues in the body language of the human trainer, who was entirely unaware that he was providing such cues. In honour of Pfungst's study, the anomalous artifact has since been referred to as the Clever Hans effect and has continued to be important knowledge in the observer-expectancy effect and later studies in animal cognition. Pfungst was an assistant to German philosopher and psychologist Carl Stumpf, who incorporated the experience with Hans into his further work on animal psychology and his ideas on phenomenology.
NLP's Clever Hans Moment Has Arrived (thegradient.pub)
>This is a blog by Ian Goodfellow and Nicolas Papernot about security and privacy in machine learning. We jointly created cleverhans, an open-source library for benchmarking the vulnerability of machine learning models to adversarial examples. The blog gives us a way to informally share ideas about machine learning security and privacy that are not yet concrete enough for traditional academic publishing, and to share news and updates relevant to the cleverhans library.
While writing OP, I was curious enough about how they could solve Clever Hans that I went and read the original Pfungst book. I recommend it: his thoroughness & cleverness in studying Clever Hans (as well as the history of previous 'talking animals') ought to be better known than Clever Hans himself, IMO. (Brief review: https://gwern.net/review/book#clever-hans-pfungst-2011 )
