Summarization using LLM and measuring the performance with ROUGE— Part 1 | by Kunal Gandhi | Oct, 2023

LLM (Huge Language Fashions) are the first focus of proper now’s AI world notably inside the house of Generative AI. On this text, we’ll try few LLMs from hugging-face by the use of in-built Pipeline and might measure the effectivity of each model by the use of ROUGE.

Summarization — So there’s 2 strategy to hold out summarization.

1. Abstractive Summarization — Proper right here we try to create a summary that represents the purpose and seize the essence of the doc. That’s arduous to realize as we might need to create new phrases and recreate sentences that aren’t present inside the doc which can create grammatical and semantic factors
2. Extractive Summarization — Extractive summarization selects and extracts full sentences from the provision textual content material to create the summary. It doesn’t generate new sentences nonetheless reasonably chooses sentences that are most likely probably the most informative or marketing consultant of the content material materials.

Hugging Face transformers try to hold out abstractive summarization. Let’s come to the aim.

First it’s important to import following libraries,

# to load the dataset
from datasets import load_dataset
# to create summarization pipeline
from transformers import pipeline
# to calculate rouge score
from rouge_score import rouge_scorer
import pandas as pd

Please arrange libraries by the use of pip when you don’t already have them.

Now let’s load dataset that we going to utilize to measure the effectivity of LLMs.

xsum_dataset = load_dataset("xsum", mannequin="1.2.0") 
xsum_sample = xsum_dataset["train"].select(differ(5))
present(xsum_sample.to_pandas())

As you’ll be capable to see dataset have 3 columns.

• doc: Enter data article.
• summary: One sentence summary of the article.
• id: BBC ID of the article.

Yow will uncover additional about this dataset here.

Let’s create summarization pipeline and create summary passing doc.

summarizer_t5 = pipeline(
course of="summarization",
model="t5-small",
) 
outcomes = summarizer(xsum_sample["document"],min_length=20,max_length=40,truncation=True)
# convert to pandas df and print
opt_result = pd.DataFrame.from_dict(outcomes).rename({"summary_text": "generated_summary"}, axis=1).be a part of(pd.DataFrame.from_dict(xsum_sample))[
["generated_summary", "summary", "document"]
]   
present(opt_result.head())

Pipeline takes primarily three arguments. Model, course of and tokenizer. Proper right here we’re using the default tokenizer.

We’re passing the minimal measurement as 20 and the utmost measurement for summary is 40.

Now measure the effectivity by calculating ROUGE Ranking.

ROUGE stands for “Recall-Oriented Understudy for Gisting Evaluation.” It’s a metric designed to measure the usual of summaries by evaluating them to human reference summaries. ROUGE is a gaggle of metrics, with most likely probably the most typically used one being ROUGE-N, which measures the overlap of N-grams (contiguous sequences of N phrases) between the system-generated summary and the reference summary.

Let’s calculate ROUGE — 1 for the subsequent occasion,

Reference Summary — Local weather is scorching proper right here

Generated Summarty — Local weather could also be very scorching proper right here

Calculate F1 Ranking using precision and Recall. F1 Ranking may very well be 0.88 .

We are going to calculate ROUGE — 2,ROUGE — 3 … ROUGE — N using bi-gram,tri-gram and N-grams.

ROUGE-L: Measures the longest widespread subsequence between the system and reference summaries. This metric is way much less delicate to phrase order and should seize semantic similarity.

def calculate_rouge(data):
scorer = rouge_scorer.RougeScorer(['rouge1', 'rouge2', 'rougeL'], use_stemmer=True)
data["r1_fscore"] = data.apply(lambda row : scorer.score(row["summary"],row["generated_summary"])['rouge1'][2], axis=1)
data["r2_fscore"] = data.apply(lambda row : scorer.score(row["summary"],row["generated_summary"])['rouge2'][2], axis=1)
data["rl_fscore"] = data.apply(lambda row : scorer.score(row["summary"],row["generated_summary"])['rougeL'][2], axis=1)
return data
score_ret=calculate_rouge(opt_result)
print("ROUGE - 1 : ",score_ret["r1_fscore"].indicate())
print("ROUGE - 2 : ",score_ret["r2_fscore"].indicate())
print("ROUGE - L : ",score_ret["rl_fscore"].indicate())

I’ve tried 2 pre-trained model for the summarization.

1. T5 — Small
2. fb/bart-large-cnn

These are pre-trained fashions. We are going to further fine-tune these fashions to work increased. Yow will uncover a list of fashions on the market for summarization duties on HuggingFace here.

Whereas ROUGE is a useful instrument, it has its limitations. As an illustration, it doesn’t take into consideration the fluency and coherence of the summary. It focuses on phrase overlap, which suggests a summary can acquire a extreme ROUGE score even when it’s not very readable.

Please uncover the code inside the git repo .