A Pretraining Strategy to Improve Faithfulness in Abstractive Text Summarization

A core dilemma of abstractive text summarization is to ensure that generated summaries are faithful to the relevant source documents, that is the degree to which the generated summary accurately reflects the content of the original text. Recent studies addressed the problem of faithfulness in abstractive text summarization from various aspects and they achieved significant progress. However, those studies lack a critical consideration of how pretraining strategies can affect and refine faithfulness in abstractive text summarization models. To address this issue, we proposed a novel pretraining method and analyzed its impact on both faithfulness and summarization. The proposed method stimulates the model to attend more for the more important sections that correlate with faithfulness in the source text. Our experiment and analysis illustrate that the proposed method can achieve better results on various faithfulness metrics than the baseline model. We also discussed that the proposed pretraining strategies can even refine the generated summaries in terms of summarization metrics.