Benchmarking Component and Integration Testing in Microservices: Test Suites and Fault Analysis on TrainTicket

Microservices decompose systems into small, independent services, offering benefits that have led to widespread adoption. However, their distributed nature, diverse technology stacks, and independent development introduce significant challenges in ensuring system reliability. While numerous automated testing approaches have been proposed, only a few manually implemented test suites are openly available for comparison. To the best of our knowledge, no extensive benchmark test suites for component and integration testing of microservices are publicly accessible. This work addresses this gap by providing two benchmark test suites - one on the component and one on the integration level - developed within a case study on the well-established TrainTicket microservice system. The test suites were initially implemented by four independent teams and later unified by the authors, yielding 1,365 component and 210 integration tests in total. They uncover 51 faults, 8 untypical design decisions, and 1 fallback-related issue within the system. We further manually analyze the detected faults and compare them with existing fault taxonomies, identifying 11 gaps in current classifications. Additionally, we provide insights into the relation between the test case type and fault type it detected. Our insights can support future evaluations of new approaches and foster the development of more comprehensive fault models for microservice systems.