Better Fine-Tuning with Extracted Important Sentences for Abstractive Summarization

Recent work in abstractive text summarization using pre-trained transformers has achieved great results. Much of the work has been done on the model architectures and designing pre-training objectives. Models used for NLP tasks have grown bigger and increasing the model’s size doesn’t always improve the performance of the task. In this work we improve the results of an existing model by incorporating a new fine-tuning strategy which closely resembles how a human would summarize by a piece of text i.e. by focusing on important fragments of the text. We used PEGASUS model to work on our fine-tuning strategy. We fine-tune the model with varying ratios of important sentences in a phased manner and achieve improved ROUGE scores with an additional few thousand steps of fine-tuning. We fine-tune the model on AESLC and CNN/DailyMail dataset and achieve better ROUGE scores on both the datasets. With our proposed fine-tuning method, we get state-of-the-art results on the AESLC dataset. We also achieve better performance when fine-tuning with a limited number of examples with our fine-tuning method on AESLC dataset.