Explaining Neural Network Predictions

Have you ever wondered why a facial recognition system flagged a photo of a person with a darker skin tone as a false positive while identifying people with lighter skin tones correctly? Or why a self-driving car decided to swerve and cause an accident, instead of braking and avoiding the collision? These questions illustrate the importance of understanding why a model predicts a certain value for critical use cases. By providing explanations for a model’s predictions, we can gain insights into how the model works and why it made a specific decision, which is crucial for transparency, accountability, trust, regulatory compliance, and improved performance.

In this chapter, we will explore neural network-specific methods for explaining model predictions. Additionally, we will discuss how to quantify the quality of an explanation method. We will also discuss the challenges and limitations of model explanations and how to evaluate their effectiveness...