Explainable AI (XAI) in NLP

The increasing availability of large language models (LLMs) has brought the trade-off between the accuracy and interpretability of a model’s output to the forefront. The biggest challenge in explainable artificial intelligence (XAI) studies is dealing with the large number of layers and parameters present in deep neural models. So, how are we going to solve this problem? Will we be able to find a way to understand how a deep model makes a decision? The simple answer is no, but the other answer is somewhat.

In this chapter, we will only approach this problem in terms of Transformers. It will be examined from two perspectives. First of all, the self-attention mechanism, which is the most relevant part of the Transformer architecture regarding explainability, will be examined. This mechanism can make how the Transformer model processes an input understandable to humans. Various attention visualization tools will be used for this purpose by providing...