An Annotation Scheme for Developing Prototype Automated Facilitaion Software

We propose a novel annotation scheme for automatically facilitated online discussions based on the Issue-Based Information System (IBIS) approach. Automated facilitation of online discussions has advantages such as less influence on participants' relationships and fewer restrictions on time and place. We develop the automated facilitation system as a first step to create a large-scale consensus support system. The discussion data created by our proposed annotation scheme is used for training a machine learning model in the automated facilitation software. The process of defining our annotation scheme is as follows: 1) Human facilitators conducted experimental online discussions on the basis of the IBIS idea. 2) All the discussion data were annotated into six elements. We put ID numbers on all labeled data for indicating posting connections. 3) We tested the facilitator program's ability to judge each element. This process was repeated while improving the annotation definition. The core of our annotation definition was how to distinguish Issue and Con components. We solved this problem by defining Issues as questions able to be answered by ideas. From results of 38 annotations of experimental online discussions, the reliability of agreement between three annotators was Fleiss' kappa=0.66, which means "substantial agreement" among annotators.