An Empirical Evaluation of Structured Argumentation Using the Toulmin Argument Formalism

Some structured argumentation tools employ the Toulmin argument formalism, but no research has been performed testing this formalism's effect on argument evaluation or communication. An experiment was conducted to address this need by assessing: 1) if the process of generating Toulmin structures impacted participant (re)assessment of the soundness of an argument presented in an article and 2) if other participants thought that the structured representations adequately reflected the written argument. Results were mixed. First, generating Toulmin structures did impact the assessment of argument soundness. This was noteworthy given that participants were professionals representing the population of interest and that a weak manipulation and small sample size were used in the experiment. However, the effect was limited to the article where the argument was poorly aligned with the Toulmin formalism, and second, participants reviewing these structures found them to be less sound than the argument presented in the article itself. More generally, participants did not find it easy to generate Toulmin structures. Greater perceived difficulty in structure generation (and not generation time) was significantly correlated with the amount of change in the participants' soundness ratings, suggesting the mediating role of cognitive effort on reassessment. Generated structures varied greatly. Structures that had more total elements were easier to understand and were given better soundness ratings. These findings suggest that one needs to be cautious of the claimed value of the structured argumentation tools employing the Toulmin formalism without additional empirical research, demonstrating whether and how they can be effective cognitive aids