A quick overview of evaluation

Before we look at how different statistical techniques work, we have to have a way to measure their performance, and there are a couple of important considerations that we should review first. The first consideration is the metric or score that we assign to the system’s processing. The most common and simple metric is accuracy, which is the number of correct responses divided by the overall number of attempts. For example, if we’re attempting to measure the performance of a movie review classifier, and we attempt to classify 100 reviews as positive or negative, if the system classifies 75 reviews correctly, the accuracy is 75%. A closely related metric is error rate, which is, in a sense, the opposite of accuracy because it measures how often the system made a mistake. In this example, the error rate would be 25%.

We will only make use of accuracy in this chapter, although there are more precise and informative metrics that are actually...