Rouge Evaluation Metrics

Understanding ROUGE Evaluation Metrics for Summarization

When evaluating the performance of a summarization model, such as BERTSUM, it's essential to have a standardized method for measuring how well generated summaries align with human-written reference summaries. This is precisely where ROUGE (Recall-Oriented Understudy for Gisting Evaluation) metrics come into play.

ROUGE is a set of metrics used to automatically evaluate the quality of a generated summary by comparing it to one or more reference summaries, typically created by humans. These metrics were introduced by Chin-Yew Lin in the paper "ROUGE: A Package for Automatic Evaluation of Summaries."

Types of ROUGE Metrics

ROUGE encompasses several evaluation metrics, each focusing on different aspects of summary comparison:

• ROUGE-N: Measures the overlap of n-grams between the generated and reference summaries.

• ROUGE-L: Measures the longest common subsequence (LCS) between the generated and reference summaries.

• ROUGE-W: A weighted LCS-based metric, giving more importance to contiguous matches.

• ROUGE-S: Measures the overlap of skip-bigrams (pairs of words that may not be adjacent).

• ROUGE-SU: Measures skip-bigrams along with unigrams.

For most summarization tasks, ROUGE-N and ROUGE-L are the most commonly used and informative metrics.

ROUGE-N (N-Gram Overlap)

ROUGE-N quantifies the overlap of n-grams (sequences of N words) between the generated summary and the reference summary. It is typically reported in terms of recall, precision, and F1 score.

The general formula for ROUGE-N Recall is:

$$ \text{ROUGE-N Recall} = \frac{\text{Number of overlapping n-grams}}{\text{Total n-grams in reference summary}} $$

Where 'N' can be 1, 2, 3, and so on.

• ROUGE-1: Focuses on unigram (single word) overlap.

• ROUGE-2: Focuses on bigram (pair of words) overlap.

Example:

Consider the following:

• Reference Summary: The cat sat on the mat.

• Generated Summary: The cat lay on the rug.

Unigrams in Reference: the, cat, sat, on, the, mat (Total: 6) Unigrams in Generated: the, cat, lay, on, the, rug (Total: 6)

Overlapping Unigrams: the, cat, on, the (Count: 4)

Therefore, ROUGE-1 Recall = 4 / 6 = 0.6667.

ROUGE-L (Longest Common Subsequence)

ROUGE-L measures the longest sequence of words that appears in both the generated and reference summaries in the same order. Importantly, these words do not need to be contiguous in either summary.

Why ROUGE-L?

• Captures Sentence-Level Structure: It considers the sequence of words, reflecting sentence structure.

• Accounts for Word Order: Unlike simple n-gram overlap, ROUGE-L inherently values the order of words.

• Useful for Abstractive Summaries: In abstractive summarization, where generated text might paraphrase or use different wording, exact n-gram matches can be rare. ROUGE-L is more robust in these scenarios.

The formulas for ROUGE-L are as follows:

Let:

• LCS(X, Y) = Length of the longest common subsequence between generated summary X and reference summary Y.

• m = Length of the reference summary.

• n = Length of the generated summary.

Then:

• ROUGE-L Recall: $$ \frac{\text{LCS}(X, Y)}{m} $$

• ROUGE-L Precision: $$ \frac{\text{LCS}(X, Y)}{n} $$

• ROUGE-L F1 Score: $$ \frac{(1 + \beta^2) \times (\text{Precision} \times \text{Recall})}{(\beta^2 \times \text{Precision} + \text{Recall})} $$

Typically, β = 1, which gives equal importance to precision and recall.

Summary of ROUGE-N vs. ROUGE-L

| Metric | Measures | Best For | | :------- | :------------------------------------- | :------------------------------------------------ | | ROUGE-1 | Overlap of individual words (unigrams) | General content coverage | | ROUGE-2 | Overlap of word pairs (bigrams) | Fluency and local coherence | | ROUGE-L | Longest common subsequence | Structural similarity and abstractive summaries |

How ROUGE Evaluates BERTSUM Summarization

After generating summaries using a model like BERTSUM (whether extractive or abstractive), ROUGE scores are used to:

1. Compare Machine-Generated Summaries: Assess the quality against human-written reference summaries.

2. Evaluate Summary Attributes: Gauge content coverage, fluency, and structural alignment with references.

3. Tune the Model: Provide feedback through precision, recall, and F1 scores to optimize the summarization model's parameters.

A high ROUGE-1, ROUGE-2, and ROUGE-L score generally indicates that the generated summary effectively captures the key information from the source text in a manner that aligns with human expectations for quality and coherence.