I heard a similar story about AI being used on MRI scans.
What the researchers hadn't realised was that the samples were coming from two different models of MRI machine and that patients with more serious symptoms / worse expected outcomes were more often being sent to be scanned on the more advanced of the two.
The AI had just detected some hidden difference between the outputs and was infering that the patients scanned on the expensive machine were more likely to have a serious condition.
What the researchers hadn't realised was that the samples were coming from two different models of MRI machine and that patients with more serious symptoms / worse expected outcomes were more often being sent to be scanned on the more advanced of the two.
The AI had just detected some hidden difference between the outputs and was infering that the patients scanned on the expensive machine were more likely to have a serious condition.