Explainable and Interpretable AI and Machine Learning | Peak Indicators
Head of AI and Machine Learning at Peak Indicators, Paul Clough, summarises the concept of explainable and interpretable AI and Machine Learning, one of the most important topics of modern machine learning applications.
Introduction
Let me begin with a quote from a 2019 policy document from the Royal Society on Explainable AI :
“As AI technologies become embedded in decision-making processes, there has been discussion in research and policy communities about the extent to which individuals developing AI, or subject to an AI-enabled decision, are able to understand how the resulting decision-making system works.”
The quote highlights at least two points about current AI technologies. Firstly, that AI technologies have become embedded in decision-making processes. These may impact all manner of areas in our daily lives, including work and leisure, and where often we may not even be aware of their existence. Secondly, that there has been discussion in research and policy communities around the extent to which people (developers of AI and those affected by the outputs of AI technologies) understand how the decision-making system works.
Increasingly, AI and ML technologies, especially artificial neural networks and deep learning methods, operate as “black boxes” where the processes by which the outputs (often highly accurate) obtained are hidden with little or no explanation or rationale. And that is problematic for many reasons, not least because of the introduction of new regulations to govern the use of data-driven methods and AI technologies(e.g. the 2018 European General Data Protection Regulation that provides data subjects with the right to an explanation of algorithmic decisions or the Association for Computing Machinery statement on algorithmic transparency and accountability) and recent concerns around algorithmic bias and discrimination on the basis of certain demographics [ 1, 2]. In addition, in some highly regulated domains, such as banking and insurance, the audit and explainability of decisions is mandatory.
The notions of explainability and interpretability are often used interchangeably and refer to the degree to which a human can understand the cause of a decision. These concepts sit within the field of understandability or intelligibility that are vital to improving transparency. And this provides perhaps the greatest motivation for explainable systems -enabling greater transparency can improve users’ trust: “if the users do not trust a model or a prediction, they will not use it” [ 3]. Interpretability is therefore essential for increasing human trust and acceptance of AI and machine learning. An example of using explanations to build trust is Facebook where users can tap on posts and ads in the News Feed to see why they are shown.
Further benefits of making AI processes explainable and more transparent include[1]:
- Giving users confidence in the system*Safeguarding against bias
- Meeting regulatory standards or policy requirements
- Improving system design
- Assessing risk, robustness, and vulnerability
- Understanding and verifying the outputs from a system
- Autonomy, agency, and meeting social values
Given the benefits of explainable AI it is not surprising that problems addressing operational needs, regulatory compliance and public trust and social acceptance are amongst the often-cited use cases utilising such methods and tools.
Understanding Models and Providing Explanations
Recently there has been a focus on explainable AI-sometimes referred to as AI -a title being given to: “Tools and frameworks to deploy interpretable and inclusive machine learning models”. This aims to develop approaches for helping to understand and explain machine learning models to improve transparency and aid understanding, especially predictive models, and why they produce the results they do [ 4, 5]. For example, Figure 1 shows the DALEX (moDel Agnostic Language for Exploration and eXplanation)software library, a set of tools and methods for exploring and explaining predictive based on the principles of Explanatory Model Analysisor EMA [ 6]. This provides a useful framework in which to understand the different approaches that exist for explaining AI and machine learning models. In practice, however, there is a balance to be struck against machine learning performance versus interpretability [ 7], i.e. deep learning methods can perform very effectively, but are “black boxes” as the internal decision-making processes are highly obscured.
There are many methods to developing explainable and intelligible models and various ways have been proposed of organising these [4, 5, 6]. Generally, two key approaches are:(i) build using interpretable models(also called in-model), such as logistic regression or decision trees, that are easier to understand than “black box” methods; and (ii) provide the functionality to explain a given model and/or prediction after building the model (also called post-model). These methods are typically model-agnostic and independent from the methods used to build a model. This is particularly important when seeking to explain “black box” models, such as artificial neural networks and deep learning methods.
The DALEX framework (Figure 1), like many, differentiates between global (model or dataset) and local(instance or single prediction) level information [ 6]. Global information is used to inspect the entire dataset or the model as a whole. Whereas, local (or instance) information helps us understand our model or predictions for a single row of data (specific instance/case) or a group of similar rows. The levels in the pyramid reflect different ways of understanding the model and predictions as a whole or individual instance/case.
In the case of the model (or dataset), average metrics such as RMSE and AUC can be used to quantify how good a model is and compare models. Variable or feature importance methods can be used to show what variables most affect predictions. In some cases, unimportant variables could be removed to simplify a model.
To assess how a specific variable influences model predictions, partial dependence plots could be used. For example, controlling for all other house features, what impact do longitude and latitude have on home prices (or how would similarly-sized houses be priced in different areas)? These plots are applied after the model has been built. Finally, to understand how well a model’s predictions fit with actual values then residual plots and measures could be used to assess model fit (in general). This can be valuable in identifying cases when the model makes incorrect predictions and lead to improvements in the model.
In many situations, in addition to extracting general insights from a machine learning model using global or dataset methods, you often want to break down how the model works for an individual prediction. For example, a model says a bank should not loan someone money and the bank is legally required to explain the basis for each loan rejection; or a healthcare provider wants to identify what factors are driving a patient’s risk of some disease so they can directly address those risk factors with targeted health interventions. Again, in Figure 1 various methods exist to address questions to help understand the model and explain specific predictions. For example, the popular LIME technique is model agnostic and can be used to interpret nearly any set of machine learning inputs and machine learning predictions. Other methods, such as Ceteris-paribus profiles, can be used to examine the influence of each explanatory variable, assuming that the effects of all other variables are unchanged.
Example Tools and Libraries
The Institute for Ethical Machine Learning, an organisation that helps to develop standards for machine learning, provides a repository with a curated list of open source libraries to build and deploy machine learning applications. This includes a list of over 30 tools and libraries for AI explainability, including Python libraries from Microsoft and IBM, the IBM open sourced AI Explainability 360, a new toolkit of state-of-the-art algorithms that support the interpretability and explain ability of machine learning models.
In addition, companies, such as Google and Microsoft, are providing support for commercial explainable AI functionality. For example, Google provides Explainable AI, which are tools and frameworks to deploy interpretable and inclusive machine learning models: “Explainable AI is a set of tools and frameworks to help you develop interpretable and inclusive machine learning models and deploy them with confidence. With it, you can understand feature attributions in Auto ML Tables and AI Platform and visually investigate model behavior using the What-If Tool. “Microsoft provides resources in Azure Machine Learning for model interpretability that can be used for global/local relative feature importance and global/local feature and prediction relationship.
If you would like to find out more we’d be happy to talk with you, simply give us a call or drop us an email to set it up!
Originally published at https://www.peakindicators.com on November 18, 2022.
