This article was eye rollingly heavy on moralizing ultimately to only eventually hit on the point that what the algorithms are measuring is the odds somebody will be convicted and not the odds somebody does illegal things. Appearance is known to dramatically affect your chances at trial.
I remember reading about a similar result in software that judged job applicants. The system was trained on resumes of past applicants and the hired/not-hired outcome. Unfortunately, instead of providing a neutral guide, it just learned the biases of the humans who had made past hiring decisions. Of course, the designers of the system had excluded demographic information, but the system found proxies like extra-curricular activities that correlated with race, gender or social class.
