"sparsity is a useful measure of interpretability, since humans can handle at most 7±2 cognitive entities at once "
If you are going to limit models to 9 features, or 9 combinations of features, or 9 rules, models are not going to work as well. This just seems like a very weak argument to introduce.
"A black box model is either a function that is too complicated for any human to comprehend, or a function that is proprietary"
These seem like two very different things. A false equivalence is then made with many arguments against proprietary models being made as if they apply to complex models.
"It is a myth that there is necessarily a trade-off between accuracy and interpretability....However, this is often not true"
It's a bit of a straw man to suggest that there is "necessarily" a trade-off, as there are surely cases where there is not. One could say that there is "often" a trade-off between accuracy and interpretability. I don't think there are many people out there with the naive view that one should never do feature engineering.
"If the explanation was completely faithful to what the original model computes, the explanation would equal the original model"
This is just nonsensical to me. The idea is not to be completely faithful, but to raise things up to another level of abstraction. The series of videos from Wired comes to mind where a concept is explained at multiple levels. https://www.youtube.com/watch?v=OWJCfOvochA
"Black box models are often not compatible in situations where information outside the database needs to be combined with a risk assessment."
This is absolutely untrue, depending on one's definition of often. The output of a model, whether it falls into this incorrect definition of a model or not, can be treated as a feature in another model. Ensemble learning exists.
COMPAS is the punching bag, but no one seems to know what it is. I haven't seen the evidence that its performance is equal to three if statements. It certainly doesn't have anything to say about machine learning in general, as it is set of expert designed rules. So, it is actually the kind of algorithm the author favors, except proprietary.
"typographical errors seem to be common in computing COMPAS.... This, unfortunately, is a drawback of using black box models"
Unclear why typographical errors only affect black box models.
The BreezoMeter case is not clear evidence of anything broader either. It is unclear whether the one error noted out of millions of predictions is from bad source data. Stretching this to concern any sort of proprietary prediction, such as mortgage ratings, is a stretch that doesn't really tell us anything.
"Solving constrained problems is generally harder than solving unconstrained problems."
This doesn't make sense to me at all. All evidence is that ML works better on constrained problems.
The idea that CORELS is somehow better, even if it comes up with a ruleset of millions of rules, doesn't make sense. The proposed workaround for this "the model would contain an additional term only if this additional term reduced the error by at least 1%" could result in the failure to create a model if no one term provided 1% on its own.
Scoring systems are useful, but it's like a single layer perceptron in the example provided. You need to consider combinations of factors to see their impact. A high X is bad, unless Y is also high and Z is is low.
The fundamental problem here is trying to limit the power of the algorithm to the power of the human mind. From the very beginning we have used computers to do things that are difficult or impossible for us to do. In some cases the answers were provably correct, but we have now reached a point where computer generated proofs are accepted. The "oracle" mode of computing will compute an answer for us, but we ultimately have to choose whether or not to accept it on the basis of the evidence, much like we do with the opinion of an expert. Simple techniques such as providing one's own test data set, and the kind of analysis done by Tetlock around Superforecasters, can go a long way to building that understanding of accuracy of predictions, such that we have a guideline for evaluating algorithms that are beyond our ability to understand.
I appreciate this thoughtful and detailed reply. I was thinking all those things in my head while reading as well, but couldn't bring myself to invest the time to address them all. I got the impression that the author wasn't someone who has a lot of experience building real-world predictive models otherwise they'd appreciate the trade-offs that need to be made sometimes to get something that works well and can be debugged/interpreted without too much trouble. Of course this isn't to say we shouldn't be striving to develop more interpretable solutions, but I don't think this paper is very helpful to due to its lack of rigor and straw-man tactics.
With an interpretable model typographical errors are obvious in the result. For example, if the system denies bail because you have four convictions, but you actually don't, then the problem is obvious. If the system denies bail with no interpretation then the typographical error goes unnoticed.
I guess I don't see that part. If the typo is in the number of convictions, wouldn't an interpretable model also be subject to that typo? An interpretable model would only consider number of convictions as one of the factors. So if you look at a model like one of the scoring models shown and there are 20-30 factors under consideration, the impact would not be any more apparent than it would be from reviewing the input data. Like if it said a person has zero convictions and allowed bail, but they had four convictions, it wouldn't be obvious from the result that there was a typo somewhere.
If you are going to limit models to 9 features, or 9 combinations of features, or 9 rules, models are not going to work as well. This just seems like a very weak argument to introduce.
"A black box model is either a function that is too complicated for any human to comprehend, or a function that is proprietary" These seem like two very different things. A false equivalence is then made with many arguments against proprietary models being made as if they apply to complex models.
"It is a myth that there is necessarily a trade-off between accuracy and interpretability....However, this is often not true" It's a bit of a straw man to suggest that there is "necessarily" a trade-off, as there are surely cases where there is not. One could say that there is "often" a trade-off between accuracy and interpretability. I don't think there are many people out there with the naive view that one should never do feature engineering.
"If the explanation was completely faithful to what the original model computes, the explanation would equal the original model" This is just nonsensical to me. The idea is not to be completely faithful, but to raise things up to another level of abstraction. The series of videos from Wired comes to mind where a concept is explained at multiple levels. https://www.youtube.com/watch?v=OWJCfOvochA
"Black box models are often not compatible in situations where information outside the database needs to be combined with a risk assessment." This is absolutely untrue, depending on one's definition of often. The output of a model, whether it falls into this incorrect definition of a model or not, can be treated as a feature in another model. Ensemble learning exists.
COMPAS is the punching bag, but no one seems to know what it is. I haven't seen the evidence that its performance is equal to three if statements. It certainly doesn't have anything to say about machine learning in general, as it is set of expert designed rules. So, it is actually the kind of algorithm the author favors, except proprietary.
"typographical errors seem to be common in computing COMPAS.... This, unfortunately, is a drawback of using black box models" Unclear why typographical errors only affect black box models.
The BreezoMeter case is not clear evidence of anything broader either. It is unclear whether the one error noted out of millions of predictions is from bad source data. Stretching this to concern any sort of proprietary prediction, such as mortgage ratings, is a stretch that doesn't really tell us anything.
"Solving constrained problems is generally harder than solving unconstrained problems." This doesn't make sense to me at all. All evidence is that ML works better on constrained problems.
The idea that CORELS is somehow better, even if it comes up with a ruleset of millions of rules, doesn't make sense. The proposed workaround for this "the model would contain an additional term only if this additional term reduced the error by at least 1%" could result in the failure to create a model if no one term provided 1% on its own.
Scoring systems are useful, but it's like a single layer perceptron in the example provided. You need to consider combinations of factors to see their impact. A high X is bad, unless Y is also high and Z is is low.
The fundamental problem here is trying to limit the power of the algorithm to the power of the human mind. From the very beginning we have used computers to do things that are difficult or impossible for us to do. In some cases the answers were provably correct, but we have now reached a point where computer generated proofs are accepted. The "oracle" mode of computing will compute an answer for us, but we ultimately have to choose whether or not to accept it on the basis of the evidence, much like we do with the opinion of an expert. Simple techniques such as providing one's own test data set, and the kind of analysis done by Tetlock around Superforecasters, can go a long way to building that understanding of accuracy of predictions, such that we have a guideline for evaluating algorithms that are beyond our ability to understand.