Sentiment and Politeness Analysis Tools on Developer Discussions Are Unreliable, but So Are People

Many software engineering researchers use sentiment and politeness analysis tools to study the emotional environment within collaborative software development. However, papers that use these tools rarely establish their reliability. In this paper, we evaluate popular existing tools for sentiment and politeness detection over a dataset of 589 manually rated GitHub comments that represent developer discussions. We also develop a coding scheme on how to quantify politeness for conversational texts found on collaborative platforms. We find that not only do the tools have a low agreement with human ratings on sentiment and politeness, human raters also have a low agreement among themselves.