Including many variations of that same problem. "Sorry, you don't qualify for ____ because our 'clever' assessment algorithm doesn't account for your particular situation." Humans with decision making power correct these kind of problems all the time.
Computers with clever algorithms can be incredibly useful, but they are still just tools. Unfortunately, given how bad we humans are with statistics, it's easy to overestimate[1] their accuracy.
In addition to the raw accuracy problems, there is a risk of prejudices being baked into algorithms. We already see this with "redlining" and other housing practices, where loans availability ends up being coded racism. With the added complexity of machine learning and other prediction and analysis methods, it is probably a lot easier to hide improper discrimination.
[1] reason #57 why statistics should be a mandatory HS math class, right after algebra
It's a hairy problem. Not understanding statistics goes both ways - and from what I've seen, it usually leads to overestimating the value of human input. People are quick to distrust or reject machine's judgment when it disagrees with their biases and generally, it's much easier to debias a machine (or code it properly in the first place) than to debias humans.
While I agree that humans may enshrine their prejudices in code which will later will turn hard to adjust, again what I fear more is a reverse scenario. A machine can be perfectly fair and people won't like it, because it fails to apply the biases they want. People often apply counter-discrimination to compensate for what they believe is unfair treatment.
A mandatory class in statistics and applied probability theory could maybe help the next generation to accept that what is fair may not look so at a first glance.
> A mandatory class in statistics and applied probability theory could maybe help the next generation to accept that what is fair may not look so at a first glance.
To be honest, I don't think this will help much since most people won't interested in it and will forget what they learn as soon as the class is over. Also, being an expert in statistics can help you better understand reality, or it can help you present data in a way that confirms your biases (even unintentionally). It's sort of like sophistry, except not quite to that degree or moral ambivalence.
edit: If you want a good example of this, see Scott Alexander show how parapsychology can actually come up with some pretty good statistics while appearing to check off all the right boxes with regard to experimental design: http://slatestarcodex.com/2014/04/28/the-control-group-is-ou...
http://www.firstcoastnews.com/story/news/crime/2015/10/19/po...
An old thread on the topic:
https://news.ycombinator.com/item?id=4185684