Ironically, the better the AI, the worse this problem is going to get. The error rate will never be zero, but the closer it gets to zero, the more the AI's judgement will be presumed to be correct by the legal system. So if you happen to be one of those unlucky souls who draws the false-positive card in an era where those are rare, you could be totally hosed.
Because our current justice system never hands out false positive cards today.
The real question should be if this is better than the false positive rate we have today, though that’s probably unknowable. At least when AI fails, it fails spectacularly.
No. Sometimes AI fails spectacularly, and sometimes it fails banally (not to be confused with benignly). But when it fails spectacularly it makes the papers. When it fails banally you never hear about it.
Today society has a healthy skepticism of AI because it's new and untried. But some day that will change. AI will get better, and the spectacular failures in particular will be reigned in. Nonetheless, it will still fail some times. Society may forget that if we don't have the spectacular failures to remind us.
> our current justice system never hands out false positive cards today.
Of course it does. The difference is that we know that humans are fallible. At the moment we still know this about machines too. My concern is what happens when the machines start to appear infallible, but still aren't.
