Scaling Code Pattern Inference with Interactive What-If Analysis

Programmers often have to search for similar code when detecting and fixing similar bugs. Prior active learning approaches take only instance-level feedback, i.e., positive and negative method instances. This limitation leads to increased labeling burden, when users try to control generality and specificity for a desired code pattern. We present a novel feedback-guided pattern inference approach, called SURF. To reduce users' labelling effort, it actively guides users in assessing the implication of having a particular feature choice in the constructed pattern, and incorporates direct feature-level feedback. The key insight behind SURF is that users can effectively select appropriate features with the aid of impact analysis. SURF provides hints on the global distribution of how each feature is consistent with already labelled positive and negative instances, and how selection of a new feature can yield additional matching instances. Its what-if-analysis contrasts how different feature choices can include (or exclude) more instances in the rest of the population. We performed a user study with 14 participants, designed with two-treatment factorial crossover. Participants were able to provide 30% more correct answers about different API usages in 20% less time. All participants found that what-if-analysis and impact analysis are useful for pattern refinement. 79% of the participants were able to produce the correct, expected pattern with SURF's feature-level guidance, as opposed to 43% of the participants when using the baseline with instance-level feedback only. SURF is the first approach to incorporate feature-level feedback with automated what-if analysis to empower users to control the generality (/ specificity) of a desired code pattern.