Understanding Metrics to Assess the Quality of Classifiers

.docx

School

Northern Virginia Community College *

*We aren’t endorsed by this school

Course

MISC

Subject

Computer Science

Date

Oct 30, 2023

Type

docx

Pages

Uploaded by Hiikaa

Understanding Metrics to Assess the Quality of Classifiers The quality of any classifier is measured in terms of True Positive, False Positive, True Negative and False Negative, Precision, Recall and F-Measure. To understand this concept, let us first consider the story of boy who shouted ‘Wolf’ to fool the villagers. The story goes as follows. The boy who cried wolf Once a boy was getting bored and thought of making fools out of fellow villagers. To have some fun, he shouted out, ‘Wolf!’ even though no wolf was in sight. The villagers ran to rescue him, but then got angry when they realized that the boy was playing a joke on them. The boy repeated the same prank a number of times and each time the villagers rushed out. They got angrier when they found he was joking. One night, the boy saw a real wolf approaching and shouted ‘Wolf!’. This time villagers stayed in their houses and the hungry wolf turned the f lock into lamb chops. Let’s make the following definitions: Here, ‘Wolf’ is a positive class and ‘No wolf’ is a negative class . We can summarize our ‘wolf-prediction’ model using a 2x2 confusion matrix that depicts all four possible outcomes as shown below. True Positive (TP): Reality: A wolf threatened. Boy said: ‘Wolf.’ Outcome: Boy is a hero. False Positive (FP): Reality: No wolf threatened. Boy said: ‘Wolf.’ Outcome: Villagers are angry at Boy for waking them up. False Negative (FN): Reality: A wolf threatened. Boy said: ‘No wolf.’ Outcome: The wolf ate all the flock. True Negative (TN): Reality: No wolf threatened. Boy said: ‘No wolf.’ Outcome: Everyone is fine. Now consider a classifier, whose task is to predict whether the image is of a bird or not. In this case, let us assume, ‘Bird’ is a positive class and ‘Not a Bird’ is a negative class. Let us suppose we have a dataset of 15,000 images having 6000 images of birds and 9000 images of anything that is not a bird. The matrix illustrating actual vs. predicted results also known as confusion matrix is given in Figure 5.46 below. Results for 15,000 Validation Images (6000 images are birds, 9000 images are not birds) Predicted ‘bird’ Predicted ‘not a bird’ Bird 5,450 True Positives 550 False Negatives Not a Bird 162 False Positives 8,838 True Negatives

Figure 5.46 Confusion matrix for bird classifier

5.7.1 True positive Those instances where predicted class is equal to the actual class are called as true positive or a true positive is an outcome where the model correctly predicts the positive class. For example in the case of our bird classifier, the birds that are correctly identified as birds are called true positive. 5.7.2 True negative Those instances where predicted class and actual class are both negative are called as true negative or a true negative is an outcome where the model correctly predicts the negative class. For example, in the case of our bird classifier there are images that are not of birds which our classifier correctly identified as ‘not a bird’ are called true negatives. 5.7.3 False positive Those instances where predicted class/answer is positive, but actually the instances are negative or a false positive is an outcome where the model incorrectly predicts the positive class. For example, in case of our bird classifier there are some images that the classifier predicted as birds but they were something else. These are our false positives. 5.7.4 False negative Those instances where predicted class is negative, but actually the instances are positive or a false negative is an outcome where the model incorrectly predicts the negative class. For example, in case of our bird classifier there are some images of birds that the classifier did not correctly recognize as birds. These are our false negatives. In simple words, predicted ‘bird’ column is considered as Positive and if the prediction is correct then cell is labeled as true positive, otherwise it is false positive. The column where prediction is ‘not a bird’ is considered as negative and if prediction is correct, the cell is labeled as true negative otherwise it is false negative as shown in Figure 5.46. 5.7.5 Confusion matrix Confusion matrix is an N × N table that summarizes the accuracy of a classification model’s predictions. Here, N represents the number of classes. In a binary classification problem, N = 2. In simple words, it is a correlation between the actual labels and the model’s predicted labels. One axis of a confusion matrix is the label that the model predicted, and the other axis is the actual label. For example, consider a sample confusion matrix for a binary classification problem to predict if a patient has a tumor or not. Tumor (predicted) No-Tumor (predicted) Tumor (actual) 18 (TP) 1 (FN) No-Tumor (actual) 6 (FP) 452 (TN) Figure 5.47 Confusion matrix for tumor prediction

Your preview ends here

Eager to read complete document? Join bartleby learn and gain access to the full version