Unsupervised Extractive Summarization with BERT

The task of document summarization became more pressing as the information volume increased exponentially from news websites to scientific writings. As such, the necessity for tools that automatically summarize written text, while keeping its meaning and extracting relevant information, increased. Extractive summarization is an NLP task that targets the identification of relevant sentences from a document and the creation of a summary with those phrases. While extensive research and large datasets are available in English, Romanian and other low-resource languages lack such methods and corpora. In this work, we introduce a new approach for summarization using a Masked Language Model for assessing sentence importance, and we research several baselines for the Romanian language including K-Means with BERT embeddings, an MLP considering handcrafted features, and PacSum. In addition, we also present an evaluation corpus to be used for assessing current and future models. The unsupervised methods do not require large datasets for training and make use of low computational power. All of the proposed approaches consider BERT, a state-of-the-art Transformer used for generating contextualized embeddings. The obtained ROUGE score of 56.29 is comparable with state-of-the-art scores and the METEOR average of 51.20 supersedes the most advanced current model.