Improving labeling quality using multiple workers
Relying on a single opinion for a subjective evaluation is risky. In some cases, labeling seems straightforward; telling a car from an airplane when labeling transportation pictures is pretty simple. But let's go back to our weather data. If we're labeling air quality as good or bad based on a measurement that's not intuitive, such as the level of particulate matter (PM25), we may find that a worker's opinion depends greatly on the advice we give them and their preconceptions. If a worker believes that a certain city or country has dirty air, they are likely to favor a bad label in ambiguous cases. And these biases have real consequences – some governments are very sensitive to the idea that their air quality is bad!
One way to combat this problem is to use multiple workers to label each item and somehow combine the scores. In the notebook section called Add another worker, we'll add a second worker...
