An overview of bias, explainability, and fairness in AI/ML

While the terms “bias,” “explainability,” and “fairness” are not specific to ML, in this section, we will explore these terms as they apply to the development and use of ML models.

Bias

Bias in AI/ML refers to tendencies or prejudices in data and algorithms that can lead to unfair outcomes. One of the most common sources of bias in ML model development is when biases exist in the training data; for example, when the data points in the training data do not fairly represent the reality or the population that the model’s predictions will serve, which we refer to as data bias. For example, using a dataset in which the data points predominantly represent only one demographic group to train a model can result in poorer performance when that model is required to make predictions based on data points that represent other demographic groups. More specifically, this is an example...