Legal perspectives
There are two doctrines in anti-discrimination law: disparate treatment, and disparate impact. Let's take a minute to look at each of these:
Disparate treatment: This is one kind of unlawful discrimination. Intentionally discriminating against ZIP codes with the hope of discriminating against race is not legal. Disparate treatment problems have less to do with the algorithm and more to do with the organization running it.
Disparate impact: This can be a problem if an algorithm is deployed that has a different impact on different groups, even without the organization knowing about it. Let's walk through a lending scenario in which disparate impact could be a problem. Firstly, the plaintiff must establish that there is a disparate impact. Assessing if there's a disparate impact is usually done with the four-fifths rule, which says that if the selection rate of a group is less than 80% of the group, then it is regarded as evidence of adverse impact. If a lender has 150 loan...