An Experiment Comparing Lifted and Delayed Variability-Aware Program Analysis

Today's software systems need to be highly flexible and managing their variability plays an essential role during development. Variability-aware program analysis techniques have been proposed to support developers in understanding code-level variability by analyzing the space of program variants. Such techniques are highly beneficial, e.g., when determining the impact of changes during maintenance and evolution. Two strategies have been proposed in the literature to make existing program analysis techniques variability-aware:(i) program analysis can be lifted by considering variability already in the parsing stage; or(ii) analysis can be delayed by considering and recovering variability only when needed. Both strategies have advantages and disadvantages, however, a systematic comparison is still missing. The contributions of this paper are an in-depth comparison of SPLLIFT and COACH, two existing approaches representing these two strategies, and an analysis and discussion of the trade-offs regarding precision and run-time performance. The results of our experiment show that the delayed strategy is significantly faster but typically less precise. Our findings are intended for researchers and practitioners deciding which strategy to select for their purpose and context.