Common NLP Evaluation Metrics

Why This Matters

When you're building NLP systems—whether for classification, translation, or text generation—you need concrete ways to measure success. These evaluation metrics aren't just numbers to report; they reveal what kind of errors your model makes, how well it balances competing goals, and whether it's actually solving the problem you care about. Understanding when to use each metric is just as important as knowing the formulas.

You're being tested on your ability to select appropriate metrics for specific tasks, interpret what different scores mean, and recognize the trade-offs between metrics like precision and recall. Don't just memorize formulas—know what each metric prioritizes, when it fails, and how it connects to real-world NLP applications like spam detection, machine translation, summarization, and information retrieval.

---

Classification Metrics: Measuring Prediction Quality

These metrics evaluate how well your model assigns correct labels to inputs. The key insight is that different errors have different costs—sometimes false positives hurt more, sometimes false negatives do.

Accuracy

• Ratio of correct predictions to total predictions—the most intuitive metric, but often the most misleading
• Fails on imbalanced datasets: a spam detector that never flags spam achieves 95% accuracy if only 5% of emails are spam
• Formula: $\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}$ where TP, TN, FP, FN represent true/false positives/negatives

Precision

• Measures how many positive predictions were actually correct—answers "when the model says yes, how often is it right?"
• Prioritize precision when false positives are costly: spam filters (don't want legitimate emails in spam) or content moderation
• Formula: $\text{Precision} = \frac{TP}{TP + FP}$

Recall

• Measures how many actual positives the model found—answers "of all the things that should be flagged, how many did we catch?"
• Prioritize recall when false negatives are costly: disease detection, fraud alerts, or safety-critical systems
• Formula: $\text{Recall} = \frac{TP}{TP + FN}$

F1 Score

• Harmonic mean of precision and recall—balances both concerns into a single number
• Use when you can't afford to sacrifice either metric and need a unified score for model comparison
• Formula: $F_1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$

Confusion Matrix

• Visual table showing all four outcome types (TP, TN, FP, FN)—the foundation for calculating precision, recall, and F1
• Reveals error patterns: are you confusing class A for class B more than the reverse? Which classes are hardest to distinguish?
• Essential diagnostic tool before diving into aggregate metrics; always examine the matrix first

---

Generation Metrics: Evaluating Text Output Quality

When models produce text (translations, summaries, responses), we need metrics that compare generated output to reference texts. These metrics capture how closely the output matches human-quality text.

BLEU Score

• Compares n-gram overlap between generated text and reference translations—the standard metric for machine translation
• Brevity penalty discourages short outputs that might achieve high precision by being overly conservative
• Ranges from 0 to 1 (often reported as 0-100); scores above 30 generally indicate understandable translations

ROUGE Score

• Measures n-gram overlap for summarization tasks—focuses on recall (did the summary capture key content?)
• Multiple variants: ROUGE-N counts n-gram matches, ROUGE-L uses longest common subsequence for fluency
• Complements BLEU: BLEU emphasizes precision, ROUGE emphasizes recall of reference content

---

Probabilistic Metrics: Measuring Model Confidence

These metrics evaluate how well your model's probability distributions match the actual data. Lower uncertainty means better predictions.

Perplexity

• Measures how "surprised" a language model is by test data—lower perplexity means more confident, accurate predictions
• Exponential of cross-entropy loss: $\text{Perplexity} = 2^{-\frac{1}{N}\sum_{i=1}^{N}\log_2 P(w_i)}$
• Standard benchmark for language models: GPT-style models report perplexity to show language understanding quality

---

Retrieval and Ranking Metrics: Evaluating Search Quality

When models rank or retrieve documents, we need metrics that evaluate the quality of the entire ranked list, not just individual predictions.

Mean Average Precision (MAP)

• Averages precision scores across all recall levels for multiple queries—captures ranking quality holistically
• Rewards relevant documents appearing early in the ranked list; penalizes good results buried on page 10
• Standard metric for information retrieval tasks like search engines and question-answering systems

---

Agreement Metrics: Measuring Annotation Quality

Before training models, you need reliable labeled data. These metrics assess whether human annotators agree, which determines dataset quality.

Cohen's Kappa

• Measures inter-annotator agreement while correcting for chance—two random annotators would agree sometimes by luck
• Interpretation: $\kappa = 1$ is perfect agreement, $\kappa = 0$ is chance-level, negative values indicate systematic disagreement
• Critical for dataset validation: low kappa suggests ambiguous annotation guidelines or inherently subjective tasks

---

Quick Reference Table

---

Self-Check Questions

1. You're building a medical diagnosis system where missing a disease is far worse than a false alarm. Which metric should you prioritize—precision or recall—and why?

2. A spam classifier achieves 98% accuracy but only 40% F1 score. What does this tell you about the dataset, and why is F1 more informative here?

3. Compare BLEU and ROUGE: which emphasizes precision vs. recall, and which would you use for a summarization task?

4. Your language model achieves perplexity of 15 on test data, while a competitor's achieves 45. Which model is better, and what does perplexity actually measure?

5. Two annotators labeled 1,000 sentences for sentiment. Their accuracy (agreement rate) is 85%, but Cohen's Kappa is only 0.60. Explain why these numbers differ and which better reflects annotation quality.