Assessing the Limits of Automated Accessibility Testing: Insights for QA Engineers and Tool Developers

This research aims to evaluate the effectiveness of automated accessibility testing tools by assessing their coverage and identifying areas where manual testing remains essential. The research process begins with the selection of widely used accessibility testing tools, informed by both existing literature and a survey conducted with Sri Lankan QA engineers. Seven popular tools Axe, Lighthouse, WAVE, A11y, Includia Accessibility Checker, Accessibility Insights and Siteimprove were chosen for evaluation. Ten websites were selected for testing, and both manual testing and automated testing were conducted using WCAG guidelines (Levels A and AA). The accessibility criteria were further categorized according to different disability types. The evaluation focused on key metrics such as True Positives (TP), False Positives (FP), Issue Coverage Ratio (ICR), and Coverage Error Ratio (CER) for each tool. The findings provide insights into which accessibility issues can be effectively addressed by automated tools and which require manual intervention. These results will guide QA engineers in making informed decisions about tool selection and manual testing practices. The research offers valuable feedback for tool developers to identify areas for improvement, suggest new features, and guide the development of future tools.