An annotation scheme for references to research artefacts in scientific publications

The extraction of mentions of research artefacts from scientific papers is a necessary precursor for multiple applications ranging from simple search for literature based on particular research artefacts to semantic analyses of the investigations described in the literature. Techniques of natural language processing like named entity and relation extraction allow to establish detailed knowledge about such artefacts. The application of supervised classifiers relies on annotated datasets in order to provide a basis for training and evaluation. In this work, we present an annotation scheme for research artefacts in scientific literature which not only distinguishes between different types of artefacts like datasets, software and materials but also allows for the annotation of more detailed information such as amount or concentration of materials. Furthermore, we present first preliminary results in terms of inter-rater reliability.