Earlier this year, in the lead up to and during H2O World, I was lucky enough to moderate discussions around applications of explainable machine learning (ML) with industry-leading practitioners and thinkers. This post contains links to these discussions, written answers and pertinent resources for some of the most common questions asked during these discussions, and answers to some great audience questions we didn’t have time to address during the live discussions.
The first of these discussions were held with Tom Aliff, Senior Vice President for Analytics Solutions Consulting at Equifax. We went over various topics related to the real-world commercial application of explainable ML and also talked about Equifax’s NeuroDecisionTM interpretable neural network. To hear more from Tom, check out the discussion by replaying the on-demand webinar.
The second panel discussion was held at H2O World 2019 in San Francisco. The participants were:
This lively panel discussion covered numerous explainable ML subjects, included perspectives from financial services and healthcare, and answered several thoughtful audience questions. You can check it out on our website under H2O World replays.
We can’t thank Agus, Marc, Taposh, Tom, and Rajesh enough for their comments, thoughts, and insights. It was great to hear how these respected professionals and leaders are thinking through problems and putting explainable ML to work! We hope you learn as much from their commentary as we did.
Although I was an active moderator in these discussions, I’ve had some more time to think about the panel questions and wanted to answer the main panel questions in this blog post format where it’s easier to share resources and links. The questions, my answers, and public resources are available below.
Explainable ML, also known as explainable artificial intelligence (XAI), is a field of ML that attempts to make the inner workings and reasoning behind the predictions of complex predictive modeling systems more transparent. It’s been defined in several ways.
Finale Doshi-Valez and Been Kim gave one of the first and broadest definitions in the field for interpretability or, “the ability to explain or to present in understandable terms to a human.” So we can see that explanation is part of the broader notion of interpretability. Later Gilpin et al. put forward that, in the context of ML, a good explanation is “when you can no longer keep asking why.” The Defense Advanced Research Projects Agency (DARPA), i.e. the inventors of the internet, have also done considerable work in XAI and have given a few discussion points on their XAI homepage.
Like most fields, explainable ML is not without criticism. For an idea of what some find objectionable, see Cynthia Rudin’s Please Stop Explaining Black Box Models for High Stakes Decisions.
(This is a well-reasoned critique from one of the brightest minds in ML, but personally, I feel it’s a bit too purist. Post-hoc explanations fit nicely into entrenched business processes, and the science is moving quickly to address some of Professor Rudin’s concerns. Also, many techniques in explainable ML are meant to provide adverse action notices for regulated decisions as mandated by the Fair Credit Reporting Act (FCRA) or the Equal Credit Opportunity Act (ECOA). Such adverse action notices have to be supplied even if the underlying model is directly interpretable.)
The main drivers in the adoption of explainable ML appear to be:
Many have asked how the E.U. GDPR will affect the practice of ML in the U.S. I recommend Andrew Burt’s How Will the GDPR Impact Machine Learning for a brief primer there. I’ve also come to understand more about how explainable ML can be used for hacking, defending, and forensically analyzing ML models. If that subject is of interest to you, consider having a look at Proposals for Model Security and Vulnerability .
Outside of deep learning (which today is mostly geared toward pattern recognition in images, sound and text), I now agree with professor Rudin when she says, “It is a myth that there is necessarily a trade-off between accuracy and interpretability.” In my own experience, in constantly-changing, low signal-to-noise problems, like those relating to many human behaviors, you may see incremental improvements in accuracy in static development and test data when using very complex ML models. However, when those same very complex ML models take a long time to be deployed or have to make decisions on new data that is somehow different from their development and test data, the incremental improvements in accuracy can vanish. In many cases, I feel practitioners would be better off using more constrained models and retraining them more often.
If you’re interested to try out constrained ML models, I recommend monotonic gradient boosting machines, now available in the highly scalable, reliable, and open-source H2O-3 and XGBoost packages. , Also check out the awesome-machine-learning-interpretability metalist for more types of directly interpretable, constrained, or Bayesian ML models (and a lot of other explainable ML software and resources).
Personally, I see a win-win-win with explainable ML.
Win 1: It is possible (see NeuroDecisionTM ) to use constrained ML models to attain higher accuracy than traditional regression approaches and to retain regulator-mandated transparency.1 This higher accuracy should lead to better ROI for commercial predictive modeling endeavors.
Win 2 : Better transparency should translate into lower operational and reputational risk for commercial ML projects. Model’s that practitioners actually understand should be less likely to make giant financial mistakes, get hacked, or be discriminatory.
Win 3: When combined with other best practices, explainable ML is just the right thing to do. ML can affect people negatively. When combined with disparate impact analysis and model management and monitoring, explainable ML can help ensure models are not accidentally or intentionally harming people.
Of course, there are some losses too. Like many other technologies, explainable ML can be used in helpful or harmful ways and that’s very important to understand. In addition to their capability to hack or attack model APIs, explainable ML techniques can also be used for “fair washing,” or making discriminatory models look non-discriminatory.
In my opinion, in my own field, there is no place where interpretability is not needed. There’s simply too much risk, and interpretable and explainable ML has just become too easy to implement. (See the awesome-machine-learning-interpretability list for dozens of freely-available interpretable and explainable ML software packages. )
ML models can be hacked. ML models can be discriminatory. ML models can be wrong. Moreover, there are bad actors out there looking to make these bad things happen. Explainable ML, along with other best practices, helps us ensure our models are behaving as expected and not harming people.
The greatest need arises anywhere ML is affecting people. The potential negative effects of ML on people is a broader subject than can be addressed in this post, and explainable ML is just one small part of the problem and solution. NP Slagle gives a great introduction to the wider concerns of conscientious data scientists in his essay, On the Responsibility of Technologists: A Prologue and Primer .
At H2O we use a lot of interpretable modeling, explainable ML, and model debugging techniques including (but not limited to):
In the future, we are looking forward to combining even more techniques and tools that increase model transparency, trustworthiness, and security, and that decrease disparate impact, to provide our customers and community with a holistic, low-risk, and human-centered ML toolkit.
There were so many helpful and insightful audience questions during these discussions we were not able to answer even half during the events. Here are some of the questions we recorded and are able to answer now. If your question is not here, please ask again!
The standard best practice for fair lending purposes is to train several models and select the most acceptable model with the least disparate impact. To retrain a model with less disparate impact, look into techniques like Learning Fair Representations . To lessen the disparate impact in model predictions, look into techniques like equalized odds post-processing.
Definitely yes. Consider the related topics of ML debugging and ML security. (There are many other potential application areas as well.) In model debugging, we can use explanations of the model mechanisms, predictions, and residuals to understand errors in our model’s predictions and correct them. In ML security we can use explanations as white-hat and forensic tools to defend against model hacking.
One could also argue interpretability, in general, provides actionable insights. If my ML model is directly interpretable, then its mechanisms might yield the same kind of actionable insights as linear model coefficients and trends.
Model-agnostic explanation techniques can be applied to nearly all types of standard ML models. See Chapter 5 of Christoph Molnar’s Interpretable Machine Learning for more information about model-agnostic explanation techniques. Additionally, decision trees, neural networks , and likely other types of models can be constrained to be monotonic, which greatly increases their interpretability. , ,
The highest standard is human judgment, as discussed in Towards a Rigorous Science of Interpretable Machine Learning. 4 Because these types of human studies can be prohibitive in terms of cost and time, our team at H2O has recommended some approaches for testing model interpretability with simulated data, comparison to pre-existing explanations, or with data and model perturbation.
Yes! Both the H2O and XGBoost variants of GBM in the H2O-3 library support monotonicity constraints using the monotone_constraints hyperparameter.
No, not when we know it does not apply. I was once told that the frequency of commercial flights as a function of age is nonmonotonic. It apparently peaks in a person’s mid-forties and is lower for younger and older people. If this is true, this would be an example of when not to use monotonicity constraints.
This is a philosophical question, but I can give my own perspective. Interpretability and trust are related, but certainly not the same thing. Interpretability alone does not enable trust. In fact, interpretability can decrease trust if the model mechanisms and predictions are found to contradict human domain expertise or reasonable expectations. Interpretability, along with other properties, such as security, fairness, and accuracy, all play into human trust of the ML models.
In terms of H2O products, H2O Driverless AI probably makes interpretability easier as it can automatically train interpretable models, provides an interactive explanation dashboard, and provides scoring artifacts for generating explanations on new, unseen data.
However, open-source H2O-3 contains many explanations and interpretability features including linear models, monotonicity constraints for GBM, Shapley explanations for GBM, and partial dependence plots. I also keep a GitHub repo containing interpretable and explainable ML examples using Python, H2O-3, and XGBoost.
Yes! The tenants: “Garbage in, garbage out” and “Trust thy data” still hold.
However, I will add two caveats here (and I’m sure there are others).
Great question! While we can’t give compliance advise, I would suggest a procedure along the lines of the following:
 Explainable Artificial Intelligence: https://www.darpa.mil/program/explainable-artificial-intelligence
 In the U.S., explanations and model documentation may be required under at least the Civil Rights Acts of 1964 and 1991, the Americans with Disabilities Act, the Genetic Information Nondiscrimination Act, the Health Insurance Portability and Accountability Act, the Equal Credit Opportunity Act, the Fair Credit Reporting Act, the Fair Housing Act, Federal Reserve SR 11-7, and the European Union (EU) Greater Data Privacy Regulation (GDPR) Article 22.
 How Will the GDPR Impact Machine Learning : https://www.oreilly.com/ideas/how-will-the-gdpr-impact-machine-learning
 Proposals for Model Security and Vulnerability : https://www.oreilly.com/ideas/proposals-for-model-vulnerability-and-security
 Gradient Boosting Machine (GBM): http://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/gbm.html
 Monotonic Constraints: https://xgboost.readthedocs.io/en/latest/tutorials/monotonic.html
 Awesome-machine-learning-interpretability: https://github.com/jphall663/awesome-machine-learning-interpretability
 On the Responsibility of Technologists: A Prologue and Primer : https://algo-stats.info/2018/04/15/on-the-responsibility-of-technologists-a-prologue-and-primer/
 “Why Should I Trust You?” Explaining the Predictions of Any Classifier : https://www.kdd.org/kdd2016/papers/files/rfp0573-ribeiroA.pdf
 A Unified Approach to Interpreting Model Predictions : https://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf
 Equality of Opportunity in Supervised Learning : https://papers.nips.cc/paper/6374-equality-of-opportunity-in-supervised-learning.pdf
 Model-Agnostic Methods : https://christophm.github.io/interpretable-ml-book/agnostic.html
 Monotonic Networks : https://papers.nips.cc/paper/1358-monotonic-networks.pdf
 Optimizing Neural Networks for Risk Assessment : https://patents.google.com/patent/WO2016160539A1/en
 Testing Model Explanation Techniques : https://www.oreilly.com/ideas/testing-machine-learning-interpretability-techniques
 Interpretable Machine Learning with Python: https://github.com/jphall663/interpretable_machine_learning_with_python
 For instance, Optimized Pre-Processing for Discrimination Prevention : http://papers.nips.cc/paper/6988-optimized-pre-processing-for-discrimination-prevention.pdf
 Machine Learning Interpretability Tutorial : https://h2oai.github.io/tutorials/machine-learning-interpretability-tutorial/