Ontological Model for Contextual Data Defining Time Series for Emotion Recognition and Analysis

One of the major challenges facing the field of Affective Computing is the reusability of datasets. Existing affective-related datasets are not consistent with each other, they store a variety of information in different forms, different formats, and the terms used to describe them are not unified. This paper proposes a new ontology, ROAD, as a solution to this problem, by formally describing the datasets and unifying the terms used. The developed ontology allows information about the origin and meaning of the data to be modeled, i.e., time series, representing both emotional states and features derived from biosignals. Furthermore, the ROAD ontology is extensible and not application-oriented, thus it can be used to store data from a wide range of Affective Computing experiments. The ontology was validated by modeling data obtained from one experiment on the AMIGOS dataset. The approach proposed in the paper can be used both by researchers who create new datasets or want to reuse existing ones, and for those who want to process data from experiments in a more automated way.


I. INTRODUCTION
Affective Computing is a research area where computer science and psychology meet. Researchers attempt to use knowledge about emotions in various types of information systems, both in recognizing emotions manifested by humans and in imitating emotions by machines [66]. Practical and research applications being used in teaching and learning, therapy, job interviews and marketing [61].
The research is very often based on datasets, and thus the more intense research in this field also results in the growth of the number of published datasets obtained from experiments related to emotion processing [68]. These datasets include biosignals such as EEG (electrocardiography), ECG (electroencephalography), GSR (galvanic skin response) or facial expressions, sometimes also with emotional states. Such data can be acquired in different ways e.g., from multiple devices with various sensors or labeling by different annotators, and in the case of emotional states also from external recognition methods. Moreover, the biosignals and emotional states are also often described with contextual data (information characterizing their origin and meaning). This information is provided in different formats, sometimes only in an unstructured form in their documentation.
An exemplary excerpt of data stored within an affectiverelated dataset, obtained during an experiment analyzing biosignals of participants taking part in a recruitment interview, is depicted in Figure 1. For the exemplary participant, Agnes, the n time series were obtained from various biosignals, of which two are presented (one representing pulse and the other representing electrical conductance). Additionally, for Agnes, the m − n time series represents the obtained emotional states, from which two are presented (happiness and neutral state). For each time series, information about its meaning and origin is also provided. Information about the meaning of the data points included in time series is depicted on the left-hand side of the figure and the information about the origin of the time series is on the right-hand side.
Such a variety of information is collected within the datasets. The datasets are stored in various formats. Additionally, there is a lack of unification of the meanings of the terms VOLUME 4, 2016   used to describe the data. This often causes various terms to be used to express the same meaning while sometimes in the various datasets the same term is used to express different meanings. All of the aspects mentioned above gives researchers difficulties as the datasets are a crucial part of research.
The datasets are (re)used in studies of various types. Research which focus on emotion recognition belong to the most typical ones in Affective Computing (but are not limited to this). There are numerous emotion recognition algorithms that differ in their input channels and modalities, output labels (or affect representation model), and classification methods. "Channel" in this context is the type of signal recorded for analysis, e.g. video, while "modality" is the type of information processed to find emotion symptoms, e.g. facial expressions. Still, regardless of the different approaches applied, all of them demand datasets to construct appropriate training or test sets. Additionally, multimodal and/or multichannel observation are seen as a solution improving the recognition accuracy [33,43]. Early (feature level) fusion, late (decision level) fusion, and hybrid fusion approaches are used to integrate multiple observations, whereby each of these introduces some additional challenges. In late and hybrid fusion, the individual classifiers might report inconsistent (or even contradictory) results. The datasets containing the achieved results (recognized emotional states) make it possible to conduct research over the inconsistency issues. Costantino Thanos in [82], considering the reuse of datasets, states that "a community of practice has to establish its own domain of discourse and choose a formalism, i.e., a knowledge representation language, in order to create its own domain-specific ontology". In our case, this statements refers to the Affective Computing community. Moreover, Thanos also identifies the need to provide the explicit lexicons defining the set of terms which refer to specific concepts, i.e. in our case concepts connected with emotion analysis and recognition.
Ontologies are formal systems of concepts used to describe numerous domains of interest [30]. They gained popularity as a tool for creating common shared conceptualizations for complicated problems (such as, for instance, medicine and health care [69] [76]). Ontologies are usually expressed using logical languages (such as OWL [64]) which allow for formulating axioms that specify sometimes very complex interrelationships between concepts. Domain ontologies describe vocabularies related to a generic domain (such as medicine, or automobiles) [32]. That distinguishes them from top-level ontologies (describing very general terms) and application ontologies (depending both on a particular domain and a task).
Ontologies in OWL consist of objects (or individuals), properties (binary relations between objects, also called roles), and classes (or concepts); objects can be instances of classes. In domain ontologies, stress is usually put on defining the properties and classes and introducing the interrelationships between them through axioms; this part of an ontology is called a terminology or a TBox. Specific applications usually extend domain ontologies by introducing objects and their properties; this part of an ontology is called an ABox.
The main objective of our work is to provide a formal, expandable model for describing affective-related datasets and confirm the applicability of the OWL (Web Ontology Language) ontology for this purpose.
The novelty of the approach is providing a formal ontological description for affective-related datasets, which: • is not application-oriented, • allows describing obtained time series (both biosignals and emotional states) with information characterizing their origin and meaning, further called contextual data, • allows unifying terms in the field, • is expandable to allow defining various aspects of data obtained within the experiments. In the paper, the formal description of an affected-related dataset in the form of an OWL ontology -called ROAD (Recording Ontology for Affective-related Datasets) -is presented. The presented first version of the ROAD ontology is focused on the meaning of time series as well as origin aspects common to various types of signals (biosignals as well as emotional states, coming from sensors but also movies or other recordings). Our intent was therefore to cover specifically the contextual data. It is, however, worth stressing that the contextual data may overlap to some extent with the experiment data.
Section II describes the methodology according to which our work was conducted. The following sections III -VII follow from the adapted methodology. Firstly, in Section III, the purpose of the ontology is defined, secondly, in Section IV, the conceptualization is presented. In Section V, implementation aspects are discussed. In Section VI, possible integration with other ontologies is analyzed, and in Section VII, evaluation of the ontology is presented. These sections are followed by the presentation of related work in Section VIII and conclusions in Section IX.

II. PLANNING THE ONTOLOGY CREATION PROCESS
Before creating the ontology, we performed several preliminary steps. The first important step was the choice of the process used for the development of ROAD.
From among the available methodologies for ontology creation, we decided to use Methontology [17]. Methontology has a set of features that made it very useful for our needs: • it is based on traditional software engineering cycles, which seems particularly well suited for authoring small-to medium-sized ontologies, • the focus here is placed in the area of knowledge acquisition and conceptualization (while in e.g. the NeOn methodology [77], it is ontology reuse, and in On-To-Knowledge [79], cooperation with human experts), • it puts stress on maintenance and documentation for making the ontology more useful for its end-users, • it has the capability of being adjusted to specific needs (such as, for instance, in [9]).
Making use of the last feature, we have altered the original Methontology process slightly. The first modification stemmed from the heightened attention we gave to the creation of competency questions. In our process, we have developed them during two phases. Their first form was built during the specification phase. After performing the conceptualization, the competency questions were refined and expressed with the use of the terms that were in agreement with the identified concepts.
In the second modification, we focused on expanding the phase of ontology implementation by creating typical scenarios of its use and by modeling them in the form of UML use cases. By introducing this expansion, we have broadened the scope of use of ROAD also toward planning new recordings that have not yet been made. An additional issue that we covered here was the creation of extension points, which make it possible for the end-users to adopt and extend the contents of the ontology toward their specific needs.

III. SPECIFICATION
According to Methontology [17], the specification phase is devoted to identifying the purpose, scope, and level of formality of the implemented ontology. These tasks were performed by the authors of the paper collectively, and the results of the first two are described in the Introduction (Section I). The scope of the ontology focuses on describing the contextual data of recordings made while carrying out Affective Computing experiments. The purpose was described by us in the form of a list that includes the benefits of using ROAD.
The assumed level of formality was high from the beginning, as only such a level assures that a user can take the advantage of automated reasoning tools in order to obtain the aforementioned benefits. The existence of tools such as Protégé [44] and reasoners such as Hermit [23] facilitates the task to a significant degree.
Additionally, a very important step performed during this phase involved conducting an extensive literature study. One of the purposes of this study was to assure that ROAD indeed fills a gap in the field of Affective Computing research, and our work does not repeat any earlier developments. The effects of the study let us confirm that the scope of the work was indeed chosen in a way that makes it unique and not covered by the up-to-date research. Identified works in the field similar to ROAD exhibited different approaches, which is discussed in detail in Section VIII.
Finally, at this stage, we also created a preliminary list of competency questions, which were to be refined at the later stages of the ontology creation process depicted in Subsection IV-2. Examples of these questions embrace: • What is the characteristic of the participants taking part in the experiment? • What kinds of time series are obtained within the experiment? • What are the activities performed by the participants during the recording of biosignals?

IV. CONCEPTUALIZATION
During the conceptualization phase, we carried out the crucial task of identifying the base concepts for the ontology. This identification was supported to a large extent by knowledge acquisition performed by us actively during the project. The knowledge acquisition was carried out with the use of three main sources of knowledge: 1) Background knowledge of the experts involved in the process of the creation of the ontology -three of the five ontology authors are actively involved in other Affective Computing projects, focusing mainly on emotion recognition of children with autism and integration of emotional states obtained from various channels and algorithms. Moreover, additional knowledge was collected from other Affective Computing researchers in the form of unstructured interviews. During these discussions, the decision was made to use a disambiguated version of terms such as life activity, channel and modality [48]. 2) Literature study -the literature study performed in the previous phase of the project also allowed us to verify the set of identified concepts against the state-of-the-art papers. 3) Analysis of existing datasets -a number of datasets were analyzed (both unimodal and multimodal) such VOLUME 4, 2016 as AM-FED [58], AMIGOS [10], ASCERTAIN [78] and DEAP [45].

1) Domain description
Conceptualization was carried out in the middle-out fashion, which means that the primary or base concepts (depicted in Figure 2) were identified first. Then, the conceptualization was enriched by augmenting it with more general and more specific concepts. The primary concept is the Experiment concept whose instances denote experiments understood as a list of activities performed by the participants. These activities are recorded to obtain various biosignals and emotional states for the purpose of emotion recognition. The list of activities performed by the participant is understood as an experiment scenario, i.e. the instances of the ActivityExecution concept are related to the use of the object property nextActivityExecution defining the order of activities. The experiment is related to the first activity execution within the established order with the object property hasScenario. Each activity execution is understood as an action performed by one or more participants and can be recorded in various ways. It means that for one activity, various RegisteredData can be obtained. It opens many possibilities of activity recording, e.g. the whole scene can be recorded, each participant can be recorded separately and moreover, various channels such as sound, video, heart rate, temperature etc. can be recorded. The main concepts of the core ontology are depicted in Figure 3.
RegisteredData has one data property defined i.e. regis-teredDataSource. This property contains the URI address where the recorded data is located. A set of registered channels is associated with each registered data. The concept RegisteredChannel is introduced to represent the channels that are recorded for the specified registered data. The Channel is defined as a medium for registration of a signal holding information on observable symptoms or recognized emotional states. RegisteredChannel is related to the Regis-teredData and Channel concepts with the object properties hasRegisteredData and hasChannel, respectively. The list of individuals representing channels is standardized and depicted in Table 1.
Within the activity, the participation of the participant can be modeled as an instance of the Participation concept.
The participation is related to the activity execution (via hasActivityExecution object property) and with the instance of the ParticipantState concept.
There are two concepts -Participant and ParticipantState -describing a participant. This is due to the fact that participants are independent of the experiments. The same participant can take part in various experiments. However, some of their features can change over time. Thus, the Participant concept represents the real person, and ParticipantState his/her state during the experiment e.g. age or appearance. The relation between ParticipantState and Participant is represented as an object property hasParticipant.
The Recording concept is introduced to model the fact that the participation of the participant within the activity is recorded on the specified channel of registered data. It means that for one participant more than one recording can exist. Each recording is related to the specified participation (via the hasParticipation object property), and the specified registered channel (via hasRegisteredChannel).
The Activity Execution concept denotes activities performed by a participant or participants. Still, two activity executions can vary by the participants, but their patterns can be the same, e.g. watching a particular movie. The Activity concept was introduced to allow defining such activity patterns. There are three types of activity patterns, defined due to the number of participants taking part in it: activities performed individually (the IndividualActivity concept), activities performed in pairs (TwoPersonsActivity) and activities performed by more than two participants (GroupActivity). The ActivityExecution concept has an activity pattern defined via the role hasActivity. If an activity execution has the TwoPersonsActivity or GroupActivity pattern defined, it can also have a proper arrangement (PersonalArrangement): for a pair of participants (PersonalTwoPersonsArrangement) or for a group of at least three participants (PersonalGroupArrangement). The personal arrangement defines people's alignment with each other. The presented concepts are depicted in Figure 4.
Taking part in an activity, each participant is in the specified state, as depicted in Figure 5. The state of the participant encompasses data properties changing over time such as age and other characteristics such as Personality or Appearance. The instance of the ParticipantState concept can be related to a specified personality by the hasPersonality object property or to a specified appearance by the hasApperance object property.
The next key concept is the TimeSeries concept which represents a single time series, understood as a set of observations x t , each one being recorded at a specific time t [8]. 1 Three views for this concept are presented, the first one -defining the time series classification -in Figure 6, the second one -defining the observed data on the basis of which the given time series is generated -in Figure 7, and the third one -defining what measure the data points relate to -in Figure 8.
The distinction of a time series is based on the distinction used in [14] and [85]. In Figure 6, the two independent divisions are presented. The first one defines if the time series is the timestamp time series (the TimestampTime-Series concept) or the epoch time series (the EpochTime-Series concept). For the timestamp time series, each data point refers to a single point in time and for the epoch time series, each data point refers to a period of time with a beginning and ending at specified points in time. The second classification is connected with the time intervals between data points. For the time series where observations are made at fixed time intervals (called regularly spaced, evenly spaced, or equally spaced), the concept Regu-larlySpacedTimeSeries is introduced. For the time series that do not conform to this condition (called irregularly spaced, unevenly spaced, or unequally spaced) we introduced the complementary concept IrregularlySpacedTimeSeries. Additionally, the four concepts (RegularlySpacedTimestampTi- meSeries, IrregularlySpacedTimestampTimeSeries, Variable-LengthEpochTimeSeries, ConstantLengthEpochTimeSeries) are defined, corresponding to timestamp and epoch time series, regularly or irregularly spaced, respectively. A single time series is obtained as a result of observation of the specified unit of observation information (concept Ob-servationInformation), which is depicted in Figure 7. Each ObservationInformation refers to three objects: recording, life activity and modality. Each time series is obtained from one or more ObservationInformations. Often, time series representing biosignals are obtained from one Observation-Information and time series representing estimated emotional state from one or more ObservationInformations, depending on the recognition process.
A life activity (the LifeActivity concept) is understood as a conscious or unconscious action of a human body, which generates a specified symptom of an emotional state, which can be further analyzed in a process of emotion recognition. A modality (the Modality concept) is a type of information on a specific observable symptom extracted from a signal that can be further analyzed to estimate an emotional state. The list of individuals representing modalities and life activities is standardized and depicted in Tables 2 and 3,  Each time series is a sequence of numerical data points in successive order. Each data point relates to some value representing the value of the measure (concept Measure) specified for the whole time series. For each individual, being an instance of the Measure concept, it is possible to define the datatype, range and unit modeled as datatype properties measureDatatype, measureRange and measureUnit. The measure unit should be expressed using a code system Unified Code of Units of Measure, which includes all units of measures being contemporarily used in international sciences, engineering, and business [50]. The single measure points at the measure name (concept MeasureName). The measure name concept is introduced to make the measure name independent of the measure range or type. The TimeSeries, Measure and MeasureName concepts are depicted in Figure 8.

2) Competency questions refinement
At the end of the conceptualization stage, the competency questions were refined to relate to the identified concepts. This refinement was also a step toward a preliminary verification of the set of identified concepts: CQ 1 -What experiments contain time series for the specified measures, which values are obtained through observation of the specified channels? CQ 2 -What time series describe the particular activity? CQ 3 -What participants take part in the specified activity?
CQ 4 -What data were registered for the specified activity?
CQ 5 -What time series describe the two-person activities within which the participants were arranged in a specified way?
CQ 6 -What are the emotion estimates achieved by observation of more than one channel for the specified experiment?
CQ 7 -What are the emotion estimates expressed in the two-dimensional model for the specified participant?
CQ 8 -What are the time series for the participants taking part in the specified experiment and having the specific appearance?
CQ 9 -What time series relate to the specified measure, which values are obtained through observation of the specified channels?
CQ 10 -Which participants have the specific personality?

V. IMPLEMENTATION
Web Ontology Language (OWL) [64] was used to represent the ROAD ontology. OWL 2 was chosen as it is the language recommended by W3C to describe classes and relations between them. The ROAD ontology was expressed in the OWL DL sub-language allowing maximum expressiveness to be achieved without losing the computational completeness of the reasoning systems, as it corresponds to description logic -a particular decidable fragment of first order logic [2].
Generally, ROAD has the expressiveness of ALCOIN (D), which provides the resulting complexity of reasoning in the NEXPTIME-complete class [55]. However, a simple modification (removal of closed lists of individuals in enumerated concepts) allows the reduction of the expressiveness to ALCIN (D) and, therefore, the complexity of reasoning to PSPACE-complete [40].
The Protégé [44] editor was used to implement ROAD as it is a free, open-source platform that provides a suite of tools to construct domain models. It allows the visualization of an ontology and its validation using several reasoners. The builtin OntoGraf tool was used to visualize the ROAD domain (Figures 3 -8 were also created with the use of OntoGraf, though the original visualizations have been altered to present only the subsumption relations and the names of the properties, and to use uniform colors). Ontology consistency was checked using HermiT reasoner [23]. Moreover, the Protégé editor provides a web version (WebProtege) supporting team cooperation. This version of Protégé was mainly used to comment the ontology by its authors.
Fragments of the ROAD ontology are presented in Manchester OWL Syntax [39] in the variant used in Protégé. This syntax is easy to read and write (does not use the mathematical symbols used in the DL syntax) and was chosen to increase the readability of the paper, including for the readers without broad knowledge of description logic.
The OWL API [38] -JAVA API for creating, manipulating and serializing OWL ontologies -was used to model instances of the ROAD ontology for the AMIGOS dataset.

A. CREATING DEFINITIONS
The proposed ontology (ROAD) contains 38 concepts related to inheritance relationships, 22 object properties and 8 datatype properties. The inheritance relationships are depicted in Figure 9.
The concepts being at the highest level of inheritance (the ones asserted as subclasses of owl:Thing, e.g. Activity, ActivityExecution or Apperance) are disjoint. Additionally, when necessary, concepts inheriting from other concepts are also disjoint at the same level of the concept hierarchy (e.g. subclasses of Activity concept are asserted as disjoint to each other), as presented in Listing 1.  The cardinalities for the properties are defined both for subjects and objects. When the cardinalities regarding the object are defined, some properties are set as functional and, to force role existence, existential quantification is used. For example, each instance of ActivityExecution is related to exactly one instance of Activity. The role hasActivity is functional (only one Activity is related to the specified ActivityExistance) and additionally the axiom presented in Listing 3 is defined.
ActivityExecution SubClassOf hasActivity some Activity When the cardinalities regarding subjects are defined, analogical rules are applied, but for the inverse role. Sometimes, when the exact cardinality is needed, the appropriate cardinality restriction (min, max or exactly) is defined, as for GroupActivityExecution and TwoPersonsActivityExecution, which is depicted in Listing 4. There is also the value restriction axiom used to define that the specific type of arrangement can be defined only for the specific type of activity execution (PersonalGroupArrangement for GroupActivityExecution and PersonalTwoPerson-sArrangement for TwoPersonsActivityExecution). The value restriction axiom for PersonalGroupArrangement is depicted in Listing 5.

B. FACILITATING THE USE OF THE ONTOLOGY
Under the topic of facilitating the use of ROAD falls the important categories of activities: documentation of the ontology, and design steps that were taken to make it easier to maintain the ontology and to adjust it for the specific needs of a user. Both areas have been identified as crucial for successful ontology development in [17].
Documentation of the ontology was a task that was approached by us from two directions. The first direction was the use of standard techniques for documenting the ontology contents. These techniques embraced the use of annotations and providing the user with online documentation. The second direction was to deliver to the users' scenarios for the use of ROAD, in order to make it clear how they should interact with the ontology during specific common tasks.

1) Documenting the ontology
In our work, we adhered to the following assumptions: • all the classes, object properties, data properties in the ontology have to be annotated, • annotation property rdfs:comment will be used for describing the entities, • language of the annotations and of the entity names is English. The presence of the annotations allowed us to generate documentation publicly available in the form of a Web page https://road.affectivese.org/.

2) Scenarios for Ontology Use
Two main scenarios of the use of the ROAD ontology were identified.
In the first scenario, an existing dataset is being described with the ROAD ontology. In other words, existing data stored in another form are being rewritten to ROAD ABox (assertional part of the ontology). In such a situation, all the needed data are known in advance, before any part of ABox is defined. It is also assumed that inference is performed after creating ABox, not during the process of its creation.
In the second scenario, which is illustrated in the form of use cases in Figure 10, a new ROAD dataset is being created. In this scenario, the known problem of a lack of knowledge is addressed. It means that the inference is performed during the ABox creation to provide the ontology user with the needed information. For example, when creating the ABox, the ontology user should know which participants are assigned to the experiment, among others, to allow creating participation only for participants assigned to the specific experiment. It can not be inferred from just checking if the pair (participant, experiment) is the instance of the specific chain complex role (e.g. inverse hasParticipant o inverse hasParticipantState o inverse hasActivityExecution o inverse hasScenario) as at this stage of ontology creation, there may not be a participant state or an activity execution defined. So the assignment axioms must be added. Such an axiom assigning a participant p1 to an experiment e1 is presented in Equation 6.
In the second scenario, there are two types of use cases, creating some entities and assigning one entity to the other. The assignment activities are separated when the entities can exist independently of the experiment creation. For example, in the ROAD ontology, participants can be created, but only some of them can take part in the specific experiment. When registered data are created, the situation is opposite and registered data are assigned to the experiment as a step within the creation process (the appropriate assignment axiom is added). There is no possibility to create registered data out of the experiment scope.

3) Extension points
The aim of the extension points is to provide the possibility of adjusting the contextual information to specific needs. There are two types of extension points. The first type of extension point -further called the user-defined property extension point -allows adding user-defined properties for some of the existing concepts. The second type of extension pointfurther called the user-defined model extension point -allows defining various models for the particular concepts.
The former extension point allows the definition of new properties, not explicitly included in the current model. Adding new properties independently by various users can make it possible for the same semantic properties to differ between experiments both with names, data type and range of values. New user-defined properties should be well documented as the presented ontology does not define their semantics. When integrating data from various experiments, the unification of such properties belongs to the user and must be done according to the provided documentation. Each concept, which can be expanded with new properties, is designed as a subclass of PropertyConcept. The definition of VOLUME 4, 2016 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.
PropertyConcept can have more than one Property defined. The relationship is achieved by the object property hasProperty. Each property represents a key-value pair via the functional hasKey and hasValue datatype properties. Each of the concepts Experiment, Activity, Activi-tyExecution, Participant, ParticipantState, RegisteredData, Recording and TimeSeries is a subclass of the PropertyConcept.
The standard extension model for ontologies is based on ontology imports. The user-defined property extension point was introduced (despite the fact that new properties can be added just by creating a new ontology, which imports ROAD) to provide the possibility to design tools based on the one common ontology, without the need to redefine it. Nevertheless, it is always possible to extend the existing ontology by imports. In the ROAD ontology, it is assumed that for some of the existing concepts, individuals should be defined as instances of their subconcepts. These concepts are denoted with an annotation property conceptType whose value is set to abstract. These are the following ones: Arrangement, PersonalTwoPersonsArrangement, Personal-GroupArrangement, Apperance, Personality, MeasureName and EmotionStateMeasureName. For all of these concepts, the idea is the same. If the role points at the individual being an instance of the abstract concept, it means that the individual is also an instance of the specified subconcept makine it possible to define the arrangement, appearance, personality or measure names according to the specified and introduced model. This extension is called a user-defined model extension point. In the ROAD ontology, an exemplary set of such models is introduced. Obviously, the existing models (defined by other authors) can also be reused, which is further discussed in Section VI. Also, the new models can be defined.
In the next paragraphs, the user-defined model extension points defined in ROAD are presented. Firstly, the one related to measure names (measure name extension point), secondly the two ones that make it possible to define the participant state in more detail (personality extension point and appearance extension point). The last one relates to the arrangement of participants within the performed activity (arrangement extension point).
Measure name extension points. Measure name extension points provide the possibility to standardize notions and introduce lexicons for measures obtained from biosignals, often used in the process of emotion recognition, as well as for measures representing emotions. In the current version of ROAD, concepts allowing the definition of names of emotions are introduced as instances of the EmotionalState-MeasureName concept. This concept is also annotated as an abstract concept and the specific emotion representation models are defined as instances of its subconcepts. There are several popular models for the representation of emotions in affective computing [31]. The first one is Ekman's model of basic emotions (happiness, anger, fear, sadness, surprise, and disgust), sometimes expanded with a neutral state if none of the six occurs [16,15]. Another popular model is the PAD model defining dimensions of emotional states, namely P-pleasantness (valence), A-arousal, and D-dominance [59,60].
Names of emotions for the Ekman model are defined as instances of the EkmanModelMeasureName concept, the neutral state is defined as an instance of NeutralStateMea-sureName and for the PAD model dimensions are defined as instances of PADModelMeasureName. The instances of EkmanModelMeasureName are depicted in Table 4.
Personality extension points. In psychology, there are several prominent traits models including Allport's trait theory, Cattell's 16 Factor Model, Eysenck's Giant Three, the Myers-Briggs Type Indicator (MBTI) and the Big Five Model [57]. In the ROAD ontology, we focus on the Big Five Model (Five Factor Model) [65], which is the most widely accepted trait model of our time. To define a personality in the Big Five model, an individual being an instance of the PersonalityBigFiveModel must be defined. This individual, representing the personality in the Big Five Model, must have five values settled -each corresponding to one of the five factors: openness, neuroticism, agreeableness, conscientiousness and extroversion -designed as data type properties ope nnessValue, neuroticismValue, agreeablenes sValue, conscientiousness and xtroversionVa lue, each taking a float value in the range <0,1>.
Appearance extension points. Two models describing appearance were introduced. The first one (ApperanceSoma-totypeModel) provides a simple version of the somatotype taxonomy introduced by W. H. Sheldon [37]. According to this taxonomy, three somatotypes are defined: ectomorphic, endomorphic and mesomorphic, each can be defined in <1,7> scale as a value of data type property: hasSomatotypeEc tomorph, hasSomatotypeEndomorph and hasSoma totypeMesomorph.
The second model is the consequence of the influence of occlusions of the face parts on emotion recognition [75,47]. The appearance occlusion model (the ApperanceO-clusionModel concept) makes it possible to define if a participant has a beard (hasBeardValue object property), moustache (hasMoustacheValue object property) or wears glasses (hasGlasses data type property taking a boolean value). The properties hasBeardValue and hasMoustacheValue can take no, some or heavy values implemented as instances of two equivalent concepts BeardValue and MoustacheValue (individuals apperanceNo, aperanceSome, apperanceHeavy).
Arrangements extension points. One arrangement model was introduced for PersonalTwoPersonsArrangement. The model implements the impersonal distance categorization introduced by E. T. Hall [34]. The model assumes that there are four zones: intimate, casual, socioconsultive and public (implemented as arrangementDistanceIntimateZone, arrangementDistanceCasualZone, arrangementDistanceSo-cioConsultiveZone and arrangemetDistancePublicZone -instances of the ArrangementDistance concept). The model ArrangementInterpersonalDistanceModel assumes that its instance is related to the appropriate value of Arrangement-Distance via the hasArrangementDistance role.
Each extension point is implemented as a separate ontology, importing the core ROAD ontology. This rule allows a user to define a new ontology that contains only these extensions which are needed in a specific application.

VI. INTEGRATION WITH OTHER ONTOLOGIES
The previous section provided a detailed description of the extension points available for the ROAD ontology. In this section, we focus on existing ontologies describing models of emotions, people personality and appearance, that we identified as possibly useful for integration with the ontology presented in the paper by using the user-defined model extension points.
In the literature, many formal models of emotions exist that can be integrated with the ROAD ontology by using the measure name extension points. The EmotionML [73], which is a W3C recommendation, is particularly noteworthy. EmotionML is a markup language designed to be usable in a broad variety of technological contexts while reflecting concepts from the affective sciences. EmotionML does not provide a single vocabulary of emotion terms, but gives users a choice to select the most suitable emotion vocabulary in their annotations. The following vocabularies are defined. For categorical descriptions, the "big six" basic emotion vocabulary by Ekman [16], an the everyday emotion vocabulary by Cowie et al. [11], and three sets of categories that lend themselves to mappings to appraisals, dimensions and action tendencies: the OCC categories [63], the categories used by Fontaine et al. [18], and the categories from the work by Frijda [20]. Three-dimensional vocabularies are provided, the pleasure-arousal-dominance (PAD) vocabulary by Mehrabian [60], the four-dimensional vocabulary proposed by Fontaine et al. [18], and a vocabulary providing a single 'intensity' dimension for such use cases that want to represent solely the intensity of an emotion without any statement regarding the nature of that emotion. For appraisal, three vocabularies are proposed: the OCC appraisals [63], Scherer's Stimulus Evaluation Checks [72], and the EMA appraisals [28]. Finally, for action tendencies, only a single vocabulary is listed, namely the one proposed by Frijda [20].
Onyx [70] is an RDF vocabulary that models emotions and the emotion analysis process itself. It can be used to represent the results of an emotion analysis service or the lexical resources involved. It includes EmotionML vocabularies or categories, dimensions and appraisals. The key concepts of the Onyx ontology are: Emotion, EmotionSet and Emotion-Analysis. The EmotionAnalysis instance contains information about the source (e.g. dataset) from which the information was taken, the algorithm used to process it, and the emotion model followed (e.g. Ekman's model). Emotion model includes EmotionCategory which is a specific category of emotion, linked through the hasEmotionCategory property; the emotion intensity via hasEmotionIntensity; action tendencies related to this emotion, or actions that are triggered by the emotion; appraisals and dimensions.
EmotiOn [80] is an ontological representation of the Plutchik's wheel of emotions model [67]. The main classes of this ontology are Emotion, Neutral and Intensity. The ontology also contains three object properties: hasIntensity, isOppositeOf, and isComposedOf. Emotion is the most important class of this ontology. It contains four subclasses: IntenseEmotion, BasicEmotion, MildEmotion, and ComplexEmotion, each containing eight subclasses for a total of 32 classes.
Human Emotion Ontology (HEO) [26] is a high level ontology for human emotions, which supplies the most significant concepts and properties that are necessary to provide accurate human emotion descriptions. The main class of HEO is Emotion, which can be described both in a discrete way, by using the hasEmotionCategory property, and in a dimensional way, by using the hasDimension property. HEO introduces two main disjoint classes for describing emotions by category: BasicEmotionCategory and Complex-EmotionCategory. Different models can be used both for expressing the basic emotions, e.g., the 6 emotions by Ekman [16], and for the complex emotions using wider emotion sets, e.g., the 48 descriptors by Cowie [11]. HEO uses the hasDimension property which includes the PAD model [60] and the four dimensions [18] to describe emotions by dimension. It has been developed in OWL description logic to take advantage of its expressiveness and its inference power in order to map the different models used in the emotion description.
EMONTO [29] is an extensible ontology that represents emotions under different categorization proposals. The key class is Emotion, which has a category (hasCategory) ac- VOLUME 4, 2016 cording to a Category class. The current version of EMONTO considers Ekman's, Cowie's, and Plutchik's emotion categorizations, which group emotions into 6, 25, and 56 (8 basic emotions) values, respectively. A class Event connects the Object, Person, and Emotion entities. An emotional Event is produced by (isProducedBy) a Person and is caused by (isCausedBy) an Object. An Event can produce several Emotions. The entities Object and Person are general classes that can connect other ontologies. The ontology provides the modality (Modality) of the information used to recognize the emotion (e.g., Gesture, Face, Posture), and the type of annotator (AutomaticAnnotator and HumanAnnotaton). Moreover, a datatype property hasIntensity is associated with the category to express the level of confidence (a float value between 0.0 and 1.0).
Personality Measurement Ontology (PMO) [1] is the part of the Personality Measurement Ontology Platform that makes it possible to automatically classify social media users' personalities into values from the Big Five model based on their posts. It was designed for the Indonesian language and follows the bottom-up approach. Five main classes correspond to the five personality traits from the personality model, i.e.: Openness, Conscientiousness, Extroversion, Agreeableness, and Neuroticism. Each class has subclasses that describe facets related to the corresponding personality trait, e.g., Fantasy, Actions and Ideas for Openness. The PMO ontology does not contain any object or datatype properties.
Pedestrian Attributes Ontology (PAO) [56] is one of three main modules of the Unified Re-ID (re-identification) system. It allows a person's appearance to be described with details such as kind of clothes and shoes as well as wearing a hat or glasses. The PAO ontology consists of main the concepts: Person, Region, Category and Attribute. Each Person is connected with many Regions by an inverse of a function partOf. There are five subconcepts of the Region, i.e., Head, Upper body, Lower body, Whole body (upper and lower) and Foot. The subconcepts of the Category are e.g. Hat, Glasses, Dress, Jeans, or Sandals. Regions and Categories, as well as Categories and Attributes, are related to each other by the relation hasA. Attributes consist of concepts such as Color, Texture and Shape.

VII. EVALUATION
In this section, we present our actions that were aimed at assessing (and, to some extent, maintaining) the quality of the ontology. We use the term "evaluation" in the broader sense here, embracing with it also verification and validation, which respectively refer to conformance to the requirements of an ontology and compliance with the real-world entities [24].
Verification and validation are crucial steps in the process of ontology engineering. Therefore, we decided to use three techniques for carrying out these tasks, each one related to slightly differing aspects of the ontology creation process.
The first subsection describes checking the ROAD ontology against specific criteria. Here, we decided to focus on the set of criteria introduced [24], and based on the general criteria for knowledge sharing technologies defined in [25]. This set of criteria embraces consistency, completeness, conciseness, expandability, and sensitiveness.
The second subsection describes a GQM-focused analysis of the final version of ROAD performed along with the guidelines of FOCA [4]. FOCA promotes the use of questionnaires that cover the main goals of ontology creation and contain questions that can be relatively easily and precisely answered by the experts who are evaluating the ontology.
The final part of our evaluation consisted in implementing a real-world ROAD usage scenario. In this scenario, we ontologically modeled one of the affective computing-related experiments described in the AMIGOS dataset.

A. ASSESSING ONTOLOGICAL CRITERIA
During our work on ROAD, we were monitoring and assessing the criteria mentioned in [24] and embracing: consistency, completeness, conciseness, expandability, and sensitiveness.
Consistency is a feature of an ontology that makes it impossible to obtain contradictory conclusions from its contents. Contrary to only semantic consistency (inability to obtain an empty set of ontology models), [24] also focuses strongly on metaphysical consistency, which refers to a lack of contradictions between the definitions contained in the ontology and the real-world meaning of the entities described within.
We monitored both kinds of consistency throughout the whole ontology life cycle (also in the vein of the Methontology approach to ontology assessment, which, according to the method, should be continuous, not occasional). During the conceptualization and implementation phases, we created a number of supplementary ontological models whose purpose was to verify whether the meaning of the concepts and roles was as intended and whether it was possible to model the real-world phenomena with the use of the introduced definitions.
As a result, we were able to keep the descriptions of the ontology entities consistent with their intended meanings, and the whole ontology semantically consistent, as the reasoning engine did not report any errors.
Completeness of an ontology is a feature that refers to whether all of the knowledge that is required is in fact present there (explicitly or implicitly).
The completeness of ROAD was to some extent monitored with the use of the aforementioned supplementary ontological models. However, the main tool which allowed us to keep ROAD complete was ontology reviews, performed on a systematic basis with reference to the ontology scope established at the beginning of our work, and to competency questions identified in the first phases.
Conciseness refers to the lack of unnecessary and spurious definitions and/or axioms in the ontology. It also aims at removing redundancies consisting in the presence of axioms that can be inferred from other axioms.
The conciseness of ROAD was assured by the ontology reviews and discussions about supplementary ontological models. A final review was also performed to assess whether all of the definitions are useful and necessary.
The only redundancies detected during the final review concerned the concepts IndividualActivityExecution, TwoP-ersonsActivityExecution, and GroupActivityExecution. These concepts have cardinality restrictions associated with them, which make them inherently disjoint. Despite this, we decided to keep the explicit disjointness between them in the ontology, in order to make it clear that this was the intention of the authors.
Expandability of an ontology is its capability to be expanded by new axioms and definitions without disrupting its original contents.
Expandability was one of the focal points of our work. It was assured by adhering to the design principles of modularity and by the identification of positional expansion scenarios. Consequently, ROAD has been supplied with various extension points, described in more detail in Section V-B3.
The usefulness of the extension points was verified during ontology creation while we developed several additional modules, among others for describing emotions in accordance with Ekman or PAD models.
Sensitiveness relates to whether small changes in the definitions result in altering large numbers of properties guaranteed earlier.
Our work on assuring sensitiveness was very closely related to expandability problems. The core concepts and roles have been included in a separate module, which assures the guaranteed features of the description used for our domain of interest. The creation of additional modules did not introduce any disruption of this original design, therefore we deemed ROAD generally insensitive to small and/or accidental changes.

B. FOLLOWING FOCA APPROACH
The FOCA [4] approach to ontology evaluation aims at streamlining the process and removing its bias by adhering to the GQM (Goal/Quality/Metrics) principles. The authors of the method picked a set of goals (taken from [13]) and matched them against the ontology metrics proposed in [83]. This matching was performed by introducing questions which are expected to be answered by an ontology expert evaluating an ontology. To make the process of evaluation easier, each of the questions has a clear and simple answering scheme, where every answer comes from the set of {0, 25, 50, 75, 100}, which resembles a Likert scale [53].
To evaluate ROAD along with the FOCA guidelines, we presented the ontology to an ontology expert who was not involved in its creation. The expert was asked to review our ontology and answer the set of questions coming from the FOCA method. The results of the evaluation are presented in Table 5. The expert decided that the ontology fully satisfied the characteristics mentioned in the large majority of questions. The exception was Question no. 3, "Did the ontology reuse other ontologies?", where the expert accepted to some extent the explanation that ROAD was made to be as universal as possible, and thus the authors were reluctant to bind it to a single higher level ontology, along with the discussion about the use of extension points.
Nevertheless, the overall score obtained with the use of the formula provided by FOCA is very high and reaches 99.8%. The method also allowed us to reevaluate a potential drawback (reuse of other ontologies), which is discussed in more detail in Section VI.

C. AMIGOS DATASET IN ROAD
The final step in evaluating the ROAD ontology was to model an actual experiment scenario. The selected experiment was one of those whose description is contained in the data collection for the AMIGOS dataset [10]. To model the selected experiment, the ROAD ontology was expanded with a userdefined property extension point. Additionally, the following user-defined model extension points were applied: PAD-ModelMeasureName, EkmanModelMeasureName and Neu-tralStateModelMeasrueName to represent emotional states, and PersonalityBigFiveModel to represent the participants' personalities. Also, new extension points were designed: one allowing to define the Positive and Negative Affect Schedule (PANAS) scale [84] -PersonalityPanasModel, one representing neurosignal measures -NeurosignalsMeasureName and one allowing to represent the attitude of the participant to watching a movie (liking and familiarity) -MovieAttitude-MeasureName. VOLUME 4, 2016 The AMIGOS dataset is designed for research on affective reactions based on neurophysiological signals and video recordings of the face and whole body. The data was collected during experiments in which participants watched videos that evoke strong emotions. The study was conducted using short and long recordings and two types of settings were also tried: individual and group. This allows analyzing the impact of the movie's duration and social context on the emotional response. In addition, AMIGOS provides information about participants' personalities and mood, which extends the analysis possibilities.
The AMIGOS dataset consists of participant profiles, neurophysiological signals, video recordings of participants and emotional state assessments.
The developers of the AMIGOS dataset collected participants' basic data, such as sex and age, and created their personality and mood profiles. This was performed using the Big Five personality trait model and the PANAS positive and negative affect scale.
Three types of neurophysiological signals, namely electroencephalogram (EEG), electrocardiogram (ECG) and galvanic skin response (GSR), were obtained from sensors placed on the participants' bodies. AMIGOS provides data originally received from the devices, as well as preprocessed and segmented, where each corresponds to a particular movie. The preprocessed EEG signal, in the form of a time series, contains 14 components coming from different channels of the device. Similarly, ECG has 2 components, while GSR has only one.
During the experiment, the participants were recorded using two cameras. From the first, placed just below the screen, HD frontal recordings of the face were obtained. The second camera was placed above the screen and used to capture RGB and depth videos covering the whole body. All recordings and time series from the neurophysiological signals were precisely synchronized.
Data on the participants' emotional states were obtained using two methods: internal and external. The internal one consisted of the participants' assessment of their affective state by completing a special questionnaire, which had two versions. The first one, which was filled in at the beginning of the experiment, was a self-assessment of their level of arousal, valence and dominance, and if they experienced any of Ekman's basic emotions or a neutral state. The second questionnaire included the previous questions and added information about video liking and familiarity. The participants completed it after each viewing. The data was also used as an evaluation of the participant's state before the next video so that each video has an initial and final assessment.
The external method was based on the analysis of the participants' behavior by external annotators. For this purpose, the participants' face recordings were divided into 20second segments. The resulting segments were then shown in random order to three independent annotators. They made ratings of the arousal and valence levels.
In order to collect the necessary data, two types of experiments were designed: with short videos and with long videos. As the name suggests, they differ in the duration of the recordings played, but also in the setting of the participants.
The scenario of the short videos experiment was chosen to be ontologically modeled. It involved 40 participants, where each individually watched a set of short recordings, whose duration do not exceed 250 seconds. These recordings were taken from feature-length films and were selected to evoke specific affective states. Each was classified into one of four categories: HVHA, HVLA, LVHA and LVLA referring to the quadrants of the two-dimensional model of emotion representation (where the letters V and A stand for valence and arousal, while H and L indicate high and low levels of the feature). The list of movies used in the study included 16 positions, 4 for each category. The order in which they were viewed differed for each participant.

2) AMIGOS to ROAD mapping
In this subsection, the rules for mapping the data from the AMIGOS dataset to the ROAD ontology are described.
Activity. Watching a single video is mapped to a separate activity as an instance of the concept IndividualActivity. Available data about a particular video are stored as individuals of the concept Property, that in the attribute hasKey saves which information about the video it refers to, and with hasValue its value. In this way, properties with the following keys were added: Category, Dataset, Movie, Number and ID meaning, respectively, the category (a quadrant of the two-dimensional model), the source dataset, the movie from which the recording was extracted, the unique video number used in the experiments and the unique video index derived from the original dataset. These individuals are connected with the activity with the role hasProperty. Equation 8 presents this mapping on the example of video number 12.
watchingV ideo12 T ype IndividualActivity, V ideoN um12 T ype P roperty, V ideoId4 T ype P roperty, V ideoCategoryHV HA T ype P roperty, V ideoSourceDECAF T ype P roperty, V ideoSourceM ovieAirplane T ype P roperty, V idoeN um12 hasKey N umber, V ideoN um12 hasV alue 12,  (8) ActivityExecution. Watching the video within the particular participant's session is mapped to an instance of the concept IndividualActivityExecution, which is linked to the movie via the role hasActivity. In order to maintain the order of viewed videos in a particular session, the role nextActivityExecution is used, with which a chain of activities is created, where each one points to its successor.
Experiment. The experiment is mapped to an individual of the concept Experiment, and its name is stored as the value of the attribute name. Using the role hasScenario, the experiment is linked to all sessions pointing to the first in order of activity execution. Equation 9 represents the mentioned assertions.

(9)
ParticipantState. Participants were mapped to an instance of the concept Participant and linked to their gender using the role hasSex. Additionally, the participant index was stored as the value of the required attribute name. In order to store the data extracted from the questionnaires relating to personality, as well as their age, an individual of type Partic-ipantState had to be created and linked to the corresponding participant using the role hasParticipant. Their age was added as an attribute age, while the personalities, linked via the role hasPersonality, were stored as instances of concepts PersonalityBigFiveModel and PersonalityPanas-Model with attributes relevant to the features of the specific model. Equation 10 illustrates the mentioned assertions.
P 01 T ype P articipant, P 01 name "1 , P 01 hasSex sexF emale) P 01state T ype P articipantState, P 01state age 26, P 01state hasP articpant P 01 P 01panasM odel T ype P ersonalityP anasM odel, P 01panasM odel hasP ositiveAf f ect 3.7, P 01panasM odel hasN egativeAf f ect 2.0, P 01state hasP ersonality P 01panas, P 01bigF iveM odel T ype P ersonalityBigF iveM odel, P 01bigF iveM odel conscientiousnessV alue 3.9, P 01bigF iveM odel extroversionV alue 4.5, P 01bigF iveM odel neuroticismV alue 3.4, P 01bigF iveM odel agreeeablesnessV alue 6.4, P 01bigF iveM odel openessV alue 5.8, P 01state hasP ersonality P 01bigF iveM odel (10) Time Series. Three types of time series were distinguished for each activity, that is, watching a specific video by a single participant: • external annotations, where each of their components is created as a separate instance of the concept Constat-ntLengthEpochTimeSeries -2 components of the evaluation given by the first annotator (arousal, valence) -2 components of the second annotator (analogous) -2 components of the third annotator (analogous) Each time series corresponds to a CSV file, which contains a timestamp column and a data column. In the case of self-assessments, this file contains only two rows for the participants' initial and final states (with a timestamp equal to 0 and another equal to the length of the video). An exceptional case is the ratings of liking and familiarity, which are given only after watching the video, so there is only one row (with a timestamp equal to the duration of the video). The individuals corresponding to the time series are linked to the file URI using the attribute timeSeriesSource. They also link to further individuals, as described in the following paragraphs.
Measure. In order to model measures of neurophysiological signals, the special concepts NeurosignalsMeasure-Name and NeurosignalsMeasure were created as extension points, which are sub-concepts of MeasureName and Measure, respectively. In the next step, the instances of the first-mentioned concept are added. EEG signal is created as electricalImpulse, ECG as heartRate, while GSR corresponds to sweat. An instance of the concept Measure is also made, which has a specific data type added by the attribute measureDatatype. It is linked to the individual corresponding to its name using the role hasMeasureName. An example of the described mapping is shown in Equation 11. electricalImpulsP reprocessed T ype N eurosignalsM easure, electricalImpulsP reprocessed measureDatatype f loat, electricalImpuls T ype N eurosignalsM easureN ame, electricalImpulsP reprocessed hasM easureN ame electricalImpuls (11) For the time series related to participants' affective state assessments, measures defined in the appropriate emotion models were used. Arousal, valence and dominance in the PADModelMeasure model, neutrality in the Neutral-StateMeasure model, sadness, revulsion, happiness, surprise, anger and fear in the EkmanModelMeasure model. Each of these is linked via the role hasMeasureName to a corresponding measure name from those already existing in the ontology. Such as in neurophysiological signals, a data type is added, as well as a range via the attributes measure-Datatype and measureRange. The measures of arousal and valence were created twice, as they have different ranges for self-assessments and external annotations. For the time series corresponding to the self-assessments of liking and familiarity, analogous to the physiological signals, the additional concepts MovieAttitudeMeasureName and MovieAt-titudeMeasure were created, and then their instances. The attributes measureDatatype and measureRange were also added. Equation 12 illustrates these assertions using the selfassessment of fear as an example. Recording. A participant's involvement in a specific activity is translated to an individual of the concept Participation. This individual is linked to the corresponding participant by the role hasParticipantState and to the video by hasActivityExecution.
The original data from all of the participants' activities from sensors that record neurophysiological signals are stored in a single file. For this data, an individual of concept RegisteredData is added, which has the name of this file stored in the attribute registeredDataSource. In addition, an individual of concept RegisteredChannel is created, which is connected to the data using the role hasRegisteredData and to the corresponding channel, selected from the available ones, by the role hasChannel. Therefore, each participant has three corresponding instances of the concept Registered-Channel.
In order to link the recorded data to a specific activity, an occurrence of the concept Recording is created. A separate individual is added for each unique pair of instances of RegisteredChannel and Participation. It is associated with the corresponding individuals using the roles hasRegisteredChannel and hasParticipation. An additional instance of the concept Recording is also created, which only has a connection to participation. It is created for signals that do not have a data source (selfassessments and external annotations), in order to link them to the corresponding Participation. Equation 13 illustrates the mentioned assertions.  (13) Observable Information. Each recording corresponds to one type of observable information, which is saved as an instance of the concept ObservableInformation. Individuals corresponding to neurophysiological signals are also linked to modalities and life activities that already exist in the ontology using the roles hasModality and hasLifeActivity. Information corresponding to the EEG signal was connected to an individual's modalityNeu-ralActivity and lifeActivityBrainActivity, ECG to modality-HeartRate and lifeActivityHeartActivity, and GSR to modali-tySkinConductance and lifeActivityPerspiration. Equation 14 presents this translation using the EEG signal as an example. eegInf ormation T ype ObservableInf ormation, eegInf ormation hasM odality modalityN euralActivity, eegInf ormation hasRecording eeRecordingP 01vid01, eegInf ormation hasLif eActivity lif eActivityBrainActivity (14) Final connections. The created time series have an attribute timeSeriesSource with the URI of the file in which they were stored. They were also linked to their corresponding elements using the roles hasMeasure and hasObservableInformation. The ObservableInformation related to the neurophysiological signal also has modality and life activity individuals attached, and the corresponding Recording is linked to the individual with the recorded channel. In addition, EEG, ECG and external annotations have connections to the properties describing them, which are instances of the concept Property, via the role hasProperty. These specify, respectively, the information about the electrode that the EEG data comes from (key Electrode), the position of the electrode while measuring the heart rate (key Position) and the index of the person assessing the participant's state (key Annotator). An example translation of the emotion assessment is presented in Equation 15, while an example of the neurophysiological signal is shown in Equation 16. [26]. Ontologies are also used as a link between Affective Computing and other domains, e.g., Psychiatry [49].
Zhang et al. [86] introduced BIO_EMOTION, an ontologybased context model for emotion recognition which allows a modeling of user contexts, including user profile, EEG data, the situation and environment factors, as well as supports reasoning on the user's emotional state(s). The key top-level elements of the ontology consist of the Emotion, User, and Situation concepts with hasEEGFeature and hasEmotion properties. The focus of the ontology is on modeling low-level biometric features and mapping such low-level information to high-level human emotions. The ROAD ontology is not intended to support the inference of a user's emotions. It is more focused on describing emotionally annotated data with the context in which they were collected.
Horvat et al. [42] applied an ontology to improve the description of emotionally annotated databases, i.e., International Affective Picture System (IAPS) and International Affective Digitized Sounds (IADS). They also use knowlegdge from the WordNet lexical database to semi-automatically connect semantically-related tags from IAPS and IADS. The proposed ontology consists of two main concepts Stimulus and DescibingConcept. Stimulus is described by three data properties: pleasure and arousal, and resource (which represents the resource file's name and location). Describ-ingConcept subsumes all concepts derived from the IAPS and IADS keywords and WordNet which are relevant in describing the meaning of stimuli. Each Stimulus has to be connected with some DescibingConcept by a relation hasPrimaryMeaning and can have many connections by a relation hasSecondaryMeaning.
Horvat [41] proposed StimSeqOnt -an ontology for the formal description of sequences containing emotionprovoking multimedia documents. It enables the modeling of experiments from the field of Affective Computing which aim to provoke and measure certain emotions. The main concept of the StimSeqOnt ontology is the Session which can consists of one or more Sequence(s). Each Sequence contains SequenceItems, i.e., Stimuli and Pause, connected with each other in an ordered list. Each SequenceItem has its duration. Thus, it is possible to reproduce each Session exactly in the same way. However, this solution does not allow the storage of the resulting biomedical signals and user emotions in an ontology. They have to be manually collected and analyzed separately.
Bratsas et al. [6] proposed an ontological framework for an integrative description of neuroscience patterns and studies. The authors paid attention to the need of unified description of research (research group, researchers, etc.), experiments (experiment task, experimental protocol parameters, study duration, etc.) and acquisition systems (EEG, Skin Conductance Device, etc.). Each of these identified areas has its own corresponding ontology in the framework, e.g. Research Ontology and Experiment Ontology. The framework was validated on the data from the emotion recognition experiment which used EEG and emotion-provoking images to measure emotional response in participants. This approach differs from ours in a couple of aspects. The most important difference is the scope of the frameworks. The one proposed by Bratsas et al. aims at modeling all data related to a piece of research with the experimental procedure and information about the researchers, while our study focuses mostly on the description of the data that is a result of experiments. Another difference is that the framework by Bratsas et al. consists of many area-related ontologies connected with each other, while ours consists of one core ontology connected with many specialized ontologies by using extensions points.
Hastings et al. [35] proposed an approach for annotating data from neuroscience experiments with concepts from a realism-based ontology, i.e., the Emotion Ontology [36]. The authors tested their proposal on the BrainMap dataset [46], which is the largest curated database of coordinates and metadata for studies in cognitive neuroscience, including affective neuroscience. However, this solution is limited to functional neuroimaging research results only, while our approach is more general and allows a description of data collected from different bio-signals (EEG, ECG, fMRI, EDA, BVP) as well as other modalities such as facial expressions, gestures and speech. Moreover, the solution of Hastings et al. is based on the Emotion Ontology, which models emotions as discrete values not related to any known model of emotions. Our ontology supports two well-known emotions models, i.e. Ekman's and PAD models of emotions, and allows for extensions by other models, which is more suited for Affective Computing research.
SAREF is an ontology from another branch of research, the Internet of Things (IoT). Its first version focused on the issue of conserving energy within smart home environments [12]. However, SAREF evolved into an ontology describing the IoT domain in general [74], and was incorporated as an industry standard endorsed by the European Telecommunications Standards Institute (ETSI).
SAREF is built around the notion of a Device, a tangible object designed to accomplish a particular task in households. What makes SAREF related to ROAD is the fact that Devices can make measurements of specific Features of Interests that are expressed in specific units.
SAREF has many interesting characteristics, among others it can be extended to cover related areas of interest in more detail. Several such extensions exist, including those that involve collecting vital data in the form of time series (EHAW: an extension of SAREF for eHealth Ageing Well domain).
Due to the fact ROAD is built from another perspective, we did not decide to include any concepts from SAREF in our ontology. However, at the current point of development of ROAD, it is entirely possible to build a bridge between the two models with use of extension points. During future ROAD development and after gathering more experience, we might decide to expand the ontology by adding a devicefocused layer, and SAREF seems the best candidate for its foundation. VOLUME 4, 2016

IX. CONCLUSIONS
The paper presents the ROAD ontology dedicated to ontologically modeling datasets in Affective Computing. The ROAD ontology allows different types of datasets to be modeled through a specific set of features: • selected aspects of the experiment course can be modeled, • time series can represent both recognized emotional states and various measures obtained from biosignals, • a way of adjusting the ontology to the specific needs, by introducing user-defined properties, • the set of extension points allowing to model appearance, personality, arrangement and emotions in several ways, • lexicons for channels, modalities and life activities. The usefulness of these features has been verified during the case study. It proved that the ROAD ontology is a selfcontained tool and in the current version can be used to formally describe a broad range of datasets.
Nevertheless, the study also allowed us to identify the following further directions in which the ROAD ontology can be developed: 1) The ROAD ontology makes it possible to model contextual data defining time series, but the time series themselves are not ontologically modeled. The next step is to provide an ontological model allowing to define time series and relationships between data points from different time series, which is especially important for measures retrieved from ECG [19]. These new features are especially important for time-series classification algorithms, which are learning from different experiments while avoiding a manual mapping of timeseries and emotional states. 2) Providing the ROAD model for time series allows for starting works concerning modeling contextual data, including for specific data points. These data are also very important as they often influence the process of emotion recognition. For example, the quality of the obtained data points can be modeled not only for the whole time series, but also for specified periods of time or even particular data points. 3) To easily reuse datasets, it is extremely important to provide the lexicons for measure names. The current version of the ROAD ontology introduces three extension points defining emotional names in the two most popular emotion models -Ekman's and PAD -and the model for the neutral state. New extensions can be introduced for defining other emotion models (e.g. the Plutchnik model [67]), but also for various measure names that can be obtained from biosignals generated by participants. 4) In the process of obtaining time series, it is sometimes valuable to store time series derived from other ones. For example, to store both an irregularly spaced time series and the regularly spaced time series derived from it. We therefore plan to include this derivation relationship in the new version of ROAD. 5) The more extension points that are defined, the wider the range of datasets that can be modeled with the ROAD ontology. As the ROAD ontology is focused on the origin aspects common for various types of signals, the extension points also allow standardized modeling of signal-specific information such as sensor location and type. Thus, future works will also concentrate on engaging a wider group of scientists in the process of defining extension points and developing the procedure of submission and review of extension points to promote them as "standardized" extension points. With respect to the usage of the ROAD ontology in various applications, the key aspect is to provide a toolset that allows researchers who are not familiar with ontologies to easily create ROAD datasets. In this paper, we generally tried to abstract from the topic of using specific tools for building ROAD datasets. However, we are aware of the importance of the subject, and in parallel with the development of the ontology, works focused on building ontology-driven interfaces and storages for ROAD datasets are underway.
Finally, an important aspect of our works is the dissemination of the solution among researchers, as we strongly believe that ROAD can become a useful and widely used tool in the Affective Computing field. This includes the policy of publishing the subsequent versions of ROAD on the publicly available web site, https://road.affectivese.org.