Since we began our journey making tools for explainable AI (XAI) in late 2016, we’ve learned many lessons, and often the hard way. Through headlines, we’ve seen others grapple with the difficulties of deploying AI systems too. Whether it’s: 

• a healthcare resource allocation system that likely discriminated against millions of black people
• data privacy violations in a retailer’s facial recognition system
• an autonomous vehicle that was incapable of avoiding jaywalkers

AI can affect people — and in various and harmful ways. One of the most significant lessons we’ve learned is that there’s more to being responsible with AI than just explainability of ML models, or even technology in general.  

There’s a lot to consider when mitigating the wide variety of risks presented by new AI systems. We claim that responsible AI combines XAI, interpretable machine learning (ML) models, data and machine learning security, discrimination testing and remediation, human-centered software interfaces, and lawfulness and compliance. It’s been tough to hold our evolving responsible AI software to this high bar, but we also continue to make progress toward these lofty goals.   

To breakdown what we’ve learned a bit more, here are the basics of how we think responsible AI tools are most effective: 

1. They empower non-technical consumers and the public to engage and challenge AI systems.

Seemingly well-built systems can cause big problems when their results are presented to users as a final, opaque, and unappealable outcome. As is already mandated in the US consumer finance vertical, use responsible AI tools to tell your users how your AI system makes decisions and let them appeal those decisions when the system is inevitably wrong. It would be even more ideal to allow users to fully engage with AI systems, through appropriate graphical interfaces, to satisfy their basic human curiosity about how these impactful technologies work. 

2. They enable engineers and scientists to test and debug AI systems.

Users will probably never understand all the details of a contemporary AI system, but the engineers and data scientists that design and build the system should. This means building interpretable and explainable AI systems that enable exhaustive testing and monitoring for: 

• algorithmic discrimination
• privacy harms
• security vulnerabilities
• prediction quality and stability
• basic logical flaws
• software and hardware failures

Think about it like this: I don’t understand the structural engineering of the I-395 bridge in Washington DC, but along with millions of people, I put my trust in the engineers who designed and built the bridge. AI will likely be as important to us as bridges one day. Let’s start respecting the risks of AI sooner rather than later. 

3. They allow decision-makers to review and evaluate AI systems.

One of the best controls for AI systems is human oversight and review. This is why major financial firms have chief model risk officers and multiple lines of human validation , beyond data science and engineering teams, for their AI systems. Much like enabling appeals for consumers, AI systems need interfaces that empower business leaders, attorneys, and compliance personnel to evaluate the business value, the reputational risks, potential litigation, and lawfulness for an AI system. Well-meaning executives and oversight personnel can’t be accountable if they can’t get the necessary information about AI systems. So, AI systems must enable human-readable documentation or other appropriate executive review interfaces.    

If you’ve found this post helpful, I’ll be speaking about these topics later next week. I’m flattered to be included in a panel about “Tools for a More Responsible AI” with Orange Silicon Valley on October 14th, at 8:30am PT. You can register here.