A Comparative Study Of Abstractiveness In Summarization Evaluation

Text summarization is one of the challenging NLP tasks which involves both NLU and NLG capabilities to be available for a model to generate summary. Specifically, abstractive summarization involves generating summaries which are more concise and at the same time more complex than the extractive summarization. The evaluation methods for abstractive summarisation still involves human intervention. The major challenges posed in the evaluation is due to unpredictable hallucination in both reference and generated summaries. In this paper we introduce a novel measure of abstraction which could help to rate a given summarization data set, potentially benefit researchers to not only semi-automate the evaluation but also ideas to provide a scope for reward-penalize methods to detect and improve positive hallucination