Custom Named Entity Recognition VS ChatGPT Prompting: A Paleontology Experiment

Natural Language Processing is one of the most commonly applied techniques in the context of chatbots. User messages must be analyzed to detect entities and match them to specific intents in order to generate answers. Traditional approaches include Named Entity Recognizers trained over datasets relevant to the scope of the chatbot, utilizing techniques from the field of machine learning. On the other hand, Large Language Models (LLM), with OpenAI’s ChatGPT as their spearhead, have gained recently significant attention and increased popularity among both scholars and the general population. Besides their ability to produce text and respond to the users’ queries, these models can be instructed to perform certain actions through carefully designed prompts. In this paper, we perform a comparison between a custom built Named Entity Recognizer (NER) that is part of a chatbot designed for operation in the Paleontology Museum of Athens, and ChatGPT. To this end, through a prompt that defines the relevant entities and the rules that should be followed, ChatGPT is instructed to act as a NER designed for the same purpose. From the comparison over commonly employed metrics, we draw useful insights on the current limitations, capabilities and applicability of such models.