Domain Adaptation for Test Report Classification in Crowdsourced Testing

In crowdsourced testing, it is beneficial to automatically classify the test reports that actually reveal a fault - a true fault, from the large number of test reports submitted by crowd workers. Most of the existing approaches toward this task simply leverage historical data to train a machine learning classifier and classify the new incoming reports. However, our observation on real industrial data reveals that projects under crowdsourced testing come from various domains, and the submitted reports usually contain different technical terms to describe the software behavior for each domain. The different data distribution across domains could significantly degrade the performance of classification models when utilized for cross-domain report classification. To build an effective cross-domain classification model, we leverage deep learning to discover the intermediate representation that is shared across domains, through the co-occurrence between domain-specific terms and domain-unaware terms. Specifically, we use the Stacked Denoising Autoencoders to automatically learn the high-level features from raw textual terms, and utilize these features for classification. Our evaluation on 58 commercial projects of 10 domains from one of the Chinese largest crowdsourced testing platforms shows that our approach can generate promising results, compared to three commonly-used and state-of-the-art baselines. Moreover, we also evaluate its usefulness using real-world case studies. The feedback from real-world testers demonstrates its practical value.