The Impact of Data Challenges on Intent Detection and Slot Filling for the Home Assistant Scenario

Natural Language Understanding (NLU) is currently a very high-interest domain to both academia and the commercial environment, due in the largest part to the recent increased popularity of conversational systems. In this paper we focus on the home assistant application context and identify a set of language and data-related challenges that can occur in such a scenario, such as: distribution shift, missing information and class imbalance. We systematically generate datasets in the Romanian language that model these data complexities and further investigate how well two of the most prominent tools - Wit.ai and Rasa NLU - solve the tasks of intent detection and slot filling, given the considered data complexities. We perform a thorough analysis of the errors produced by both tools, and provide the most probable justification for their occurrence. We found that both tools focus extensively on the verb for identifying intents, and that antonyms, class-imbalance and certain small variations in formulation greatly impact intent and slot identification. This opens new research directions to directly address these shortcomings.