Does Subject Type Influence Software Engineering Experiment Results?

Context: A key issue when dealing with the generalization threat of software engineering experiments is to use different subject types. Objective: In this paper, we aim to investigate which subject types are used in experiments and their impact on results. Method: We have performed a systematic mapping study by manually searching experiments published from January 2014 to June 2016 in six leading software engineering conferences and journals. Results: Out of the 833 papers published in the period covered, we have identified 93 papers reporting experiments with subjects. Of these, 27 papers report experiments that have two subject types (professionals and students). We have studied the impact of subject type on the results of experiments reported in 11 of these papers. Conclusion: We have observed contradictory results. Only in some cases subject type influences experimental results. This suggests that further research is needed in order to find an explanation.