Evaluating the Impact of Data Scarcity on Model Performance in a Low-Resource Afrikaans Question Answering Model

This paper evaluates the impact of data scarcity on a generative question-answering (QA) model for Afrikaans, using a hybrid architecture that combines BERT for contextual encoding and GPT-2 for generation. Trained on a limited dataset of 1,800 Afrikaans magazine articles, the model achieved promising results, with an accuracy of 73%, a precision of 82%, a recall of 71%, an F1-score of 79%, and a Mean Reciprocal Rank (MRR) of 71%. The model demonstrated strengths in factual information retrieval, domain-specific knowledge, and culturally relevant responses, especially for frequently occurring entities. However, several limitations were observed. These included hallucinations on complex or unfamiliar topics, tokenization issues with compound words, repetitive output in longer responses, and misinterpretation of question intent. Comparative analysis with ChatGPT showed that while both models perform similarly on simple questions, the hybrid model lags significantly on more complex ones. Error pattern analysis further emphasized the model’s struggle with abstract concepts and long-context comprehension. The results underscore that data availability remains a key factor in QA model performance, particularly in answering complex or out-of-domain questions.