You can look at the gradient of the inputs. It's not just that your income is too low, but also that you don't have enough credit score, past history, etc. This basically gives you a linear model that is locally accurate at predicting what the more complicated model will do. Linear models are pretty interpretable I think.
