News Consumption Helps Readers Identify Model-Generated News

Text-generation to produce journalistic content is increasing. In parallel, there is public concern about the misuse of language generation to disseminate disinformation. In this focused evaluation study, we report on an IRB-approved experiment carried out online with non-expert news audiences to study their perception of English and Spanish news-like content generated by GPT-3 versus news stories written by journalists. We explore how readers assessed news quality in terms of linguistic coherence/readability, journalistic expertise, and credibility when they perceived articles to be model-generated vs. journalist-written. We also reveal how news consumption may impact readers’ capacity to identify generated text.