Designing a framework for high-quality labels
Annotations and reviews done by humans can be labor-intensive and susceptible to human errors and inconsistency. As such, the goal is to build datasets that are both accurate and consistent, requiring labels to meet accuracy standards as well as ensuring results from different annotators are within the same range.
These goals may seem obvious at first, but in reality, it can be very tricky to get human labelers to conform to the same opinion. On top of that, we also need to verify that a consensus opinion is not biased somehow.
Our framework for achieving high-quality human annotations consists of six dimensions. We will briefly summarize these dimensions before delving into a detailed explanation of how to achieve them:
- Clear instructions: To ensure high-quality labels, the instructions for the annotation task must be explicit and unambiguous. The annotators should have a clear understanding of what is expected of them, including...
