Labeling relevant skills in tasks: Can the crowd help?

Identifying the skills required to complete a task is an important aspect of software development. Managers perform this when triaging and assigning tasks. Developers do it when picking tasks to work on. However, this endeavor is time-consuming for experts, and difficult for newcomers. In this paper, we investigate a crowd-based approach to find the skillset of a task. Our results indicate that the crowd is able to identify the skills with a recall of 0.67 and a precision of 0.76. Further, we found no difference between novice and experienced crowd workers in identifying skills. Finally, our results suggest that tasks should be reviewed by at least four workers to leverage Voting at a 25% agreement.