✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-096

Classification Evaluation Metrics

Classification evaluation metrics measure how effectively a model assigns observations to categories. Common metrics include accuracy, precision, recall, specificity, F1-score, and ROC-AUC.

Accuracy = (TP + TN)/(TP + TN + FP + FN); Precision = TP/(TP + FP); Recall = TP/(TP + FN); Specificity = TN/(TN + FP); F1 = 2 × (Precision × Recall)/(Precision + Recall)

Explanation

Classification evaluation metrics help determine whether a classifier makes useful predictions. Their interpretation depends on the task, class distribution, and relative importance of different types of errors.

A confusion matrix organizes predictions into four categories for binary classification. True positives are positive examples correctly identified as positive. True negatives are negative examples correctly identified as negative. False positives are negative examples incorrectly classified as positive. False negatives are positive examples incorrectly classified as negative.

Accuracy measures the proportion of all predictions that are correct. Precision measures the proportion of predicted positive cases that are actually positive. Recall, also called sensitivity, measures the proportion of actual positive cases correctly identified. Specificity measures the proportion of actual negative cases correctly identified. The F1-score is the harmonic mean of precision and recall, providing a single measure that balances them.

Accuracy can be misleading when classes are imbalanced. For example, if only 1% of transactions are fraudulent, a classifier that predicts every transaction as legitimate achieves 99% accuracy but detects no fraud. In such situations, precision, recall, F1-score, precision-recall curves, and class-specific evaluation may be more informative.

The ROC curve plots true positive rate against false positive rate across classification thresholds. ROC-AUC summarizes the model's ability to rank positive examples above negative examples across thresholds. Metric selection should reflect the practical cost of false positives and false negatives, rather than relying on a single score for every problem.

Variables

SymbolMeaning

TP

True positives: positive cases correctly predicted as positive

TN

True negatives: negative cases correctly predicted as negative

FP

False positives: negative cases incorrectly predicted as positive

FN

False negatives: positive cases incorrectly predicted as negative

ROC-AUC

Area under the receiver operating characteristic curve

Example or application

A medical screening classifier examines 100 patients. It identifies 20 patients as positive, of whom 16 actually have the condition. It also correctly identifies 72 patients as negative, while 8 positive patients are missed. Therefore, TP = 16, FP = 4, TN = 72, and FN = 8. Accuracy is 88%, precision is 80%, recall is approximately 66.7%, and F1-score is approximately 72.7%.

Text

A classifier can be evaluated using a confusion matrix and several performance metrics. Accuracy measures the proportion of correct predictions. Precision measures how many predicted positives are actually positive, while recall measures how many actual positives are detected. Specificity measures how many actual negatives are correctly identified. F1-score balances precision and recall. ROC-AUC evaluates ranking performance across classification thresholds. For imbalanced datasets, accuracy alone may give a misleading impression of model quality.

Key Points

  • A confusion matrix contains TP, TN, FP, and FN.

  • Accuracy measures overall classification correctness.

  • Precision is important when false positive predictions are costly.

  • Recall is important when missing positive cases is costly.

  • Specificity measures correct identification of negative cases.

  • F1-score balances precision and recall.

  • ROC-AUC measures ranking performance across thresholds.

  • The best metric depends on class imbalance and the consequences of errors.

Applications

  • Medical disease screening

  • Email spam detection

  • Fraud detection

  • Customer churn prediction

  • Image and object classification

  • Credit risk classification

Learning Outcomes

  • Interpret a binary classification confusion matrix.

  • Calculate accuracy, precision, recall, specificity, and F1-score.

  • Explain why accuracy can be misleading for imbalanced data.

  • Choose evaluation metrics based on the costs of different prediction errors.

Practice Questions

  • What is the difference between precision and recall?

  • Why can accuracy be misleading when positive cases are rare?

  • How are false positives and false negatives represented in a confusion matrix?

Related Concepts

  • Confusion Matrix

  • Classification Threshold

  • ROC Curve

  • Precision-Recall Curve

  • Imbalanced Classification

Common Mistakes

  • Confusing precision with recall.

  • Assuming high accuracy always indicates a useful classifier.

  • Ignoring the classification threshold when comparing metrics.

  • Interpreting ROC-AUC as the accuracy at one particular threshold.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Sokolova, M., and Lapalme, G. (2009). A Systematic Analysis of Performance Measures for Classification Tasks. Information Processing and Management.
  • Powers, D. M. W. (2011). Evaluation: From Precision, Recall and F-Measure to ROC, Informedness, Markedness and Correlation. Journal of Machine Learning Technologies.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning