ProCAVIAR: Hybrid Data-Driven and Probabilistic Knowledge-Based Activity Recognition

The recognition of physical activities using sensors on mobile devices has been mainly addressed with supervised and semi-supervised learning. The state-of-the-art methods are mainly based on the analysis of the user’s movement patterns that emerge from inertial sensors data. While the literature on this topic is quite mature, existing approaches are still not adequate to discriminate activities characterized by similar physical movements. The context that surrounds the user (e.g., semantic location) could be used as additional information to significantly extend the set of recognizable activities. Since collecting a comprehensive training set with activities performed in every possible context condition is too costly, if possible at all, existing works proposed knowledge-based reasoning over ontological representation of context data to refine the predictions obtained from machine learning. A problem with this approach is the rigidity of the underlying logic formalism that cannot capture the intrinsic uncertainty of the relationships between activities and context. In this work, we propose a novel activity recognition method that combines semi-supervised learning and probabilistic ontological reasoning. We model the relationships between activities and context as a combination of soft and hard ontological axioms. For each activity, we use a probabilistic ontology to compute its compatibility with the current context conditions. The output of probabilistic semantic reasoning is combined with the output of a machine learning classifier based on inertial sensor data to obtain the most likely activity performed by the user. The evaluation of our system on a dataset with 13 types of activities performed by 26 subjects shows that our probabilistic framework outperforms both a pure machine learning approach and previous hybrid approaches based on classic ontological reasoning.


I. INTRODUCTION
Nowadays, mobile devices are increasingly capable of sensing and reasoning. This continuous evolution enables the development of intelligent context-aware applications also based on the recognition of human activities [1].
In mobile computing, activity recognition has been mainly tackled with supervised machine learning approaches on inertial sensors data [2] and more recently with semi-supervised learning [3]. While those data-driven approaches generally lead to high recognition rates considering few physical activities, their effectiveness on complex and context-dependent activities is still unclear. Moreover, discriminating activities with similar movement patterns is still a challenge. For instance, activities like walking and taking the stairs, The associate editor coordinating the review of this manuscript and approving it for publication was Kaigui Bian . or standing and standing on a bus are easily confused between them by purely statistical methods based on inertial sensors.
The context that surrounds the user (e.g., semantic location, weather, traffic condition, speed, etc.) is valuable information to mitigate these issues [4], [5]. However, especially considering semi-supervised learning settings, it is not realistic to acquire a comprehensive labeled dataset that includes the large number of possible context conditions in which activities can be performed. Moreover, since the number of context variables may be high and dynamic, the resulting machine learning model would be extremely complex.
The common-sense knowledge about the relationships between context and activities can be represented through formal models (e.g., ontologies) [6]. Hence, hybrid approaches have been proposed to refine and improve the recognition rate of purely data-driven solutions using semantic reasoning on context data. [5]. Following this promising research direction, we recently proposed CAVIAR [7], a hybrid semi-supervised and knowledge-based activity recognition framework. The main drawback of those hybrid approaches is that they are based on knowledge-based models that use rigid rules to model only the most common scenarios to exclude from the final prediction activities performed in unlikely (but still possible) context conditions. For instance, CAVIAR assumes that the user can not perform certain activities in certain semantic locations (e.g., running in an indoor environment). Such rigid rules inevitably lead to recognition errors when confronted with unusual but entirely possible real-life scenarios. Indeed, the domain of human activities is complex and the relationships between a type of activity and the context in which it can be performed are inherently uncertain [8].
The above-mentioned limitations of hybrid knowledgebased and data-driven methods highlight an open research problem: how to include uncertainty when modeling semantic relationships between activities and context? How to use this probabilistic knowledge to significantly improve the recognition rate obtained by state of the art machine learning methods?
In this paper, we tackle this research problem by proposing ProCAVIAR (Probabilistic Context-aware ActiVe and Incremental Activity Recognition), a novel activity recognition framework that combines semi-supervised learning and probabilistic context-aware reasoning. An incremental machine learning classifier is in charge of inferring from inertial sensor data a candidate probability distribution over the possible activities. A probabilistic knowledge-based reasoning engine is then used to refine the probability distribution considering context-data. The output is a context-refined probability distribution. ProCAVIAR also uses active learning to continuously improve the semi-supervised machine learning model. Differently from CAVIAR [7], that uses rigid ontological reasoning for context-refinement, ProCAVIAR proposes an original application of probabilistic ontologies, obtaining a more realistic model of common knowledge by capturing the probabilistic relationships between activities and the context in which they are performed.
Our contributions are threefold: • We propose a novel framework to model context data based on a probabilistic ontology.
• We propose a technique that combines machine learning and probabilistic reasoning to overcome known issues of activity recognition based on machine learning.
• We evaluate our system through extensive experiments, showing the potential benefits of ontological probabilistic reasoning in hybrid systems for sensor-based human activity recognition.

II. RELATED WORK
Activity recognition using mobile devices has been mainly addressed with supervised machine learning methods [1], [2], [9]. The major drawback of those approaches is that they require the acquisition of a large amount of labeled data to initialize the recognition model.
For this reason, semi-supervised learning is emerging as a powerful tool to initialize the recognition model with few labeled data points and to continuously improve it over time [3], [10]- [12]. Among the many semi-supervised techniques (e.g., self-learning, co-training), active learning is one of the most effective [13]- [16]. Existing semi-supervised activity recognition methods in mobile computing mainly recognize a restricted number of physical activities using inertial sensors data. Moreover, discriminating activities with similar motion patterns is still challenging.
The information about the context that surrounds the user (e.g., semantic location, weather condition, time of the day, etc.) can be used to significantly expand the set of considered activities and to better discriminate activities with similar motion patterns that are generally executed in different context conditions (e.g., sitting and sitting on a bus) [7].
However, the acquisition of a comprehensive training set where activities are performed in all the possible context conditions is prohibitive. The abstraction ability of common-sense knowledge can be used to generate formal models representing the relationships between context and activities [5]. Several approaches have been proposed in the literature to formally represent context data [8]. Ontologies have been preferred over other formalisms mainly for their expressive power and automatic reasoning capabilities [6], [17], [18]. There are several well-known ontologies that propose a formalism for context and activities, like SOUPA [19], MetaQ [20], and the so called foundational ontology [21]. In our work we extend the ActivO ontology proposed in [5].
The combination of ontological context reasoning tools and machine learning algorithms on inertial sensors data has been explored before. Banos et al. [22], proposed the integration of machine learning, used to derive low-level activities, with ontological reasoning, used to infer higher-level context based on the derived activities and other context sources (e.g., mood, semantic location). ProCAVIAR has a different goal since it is focused on improving the predictions, but we foresee an interesting application to that framework: our ontological probabilistic approach could replace the rigid rules of the standard ontology in [22], gaining the ability to deal with uncertainty, and hence making it more flexible.
Ontological reasoning has also been used to integrate context data derived from machine learning processes in complex industrial IoT scenarios [23]. This is a typical application of ontologies, particularly useful when data is gathered by different sources and organizations. However, this type of combination of the two technologies differs from ours, since its focus is not on improving the activity recognition rate.
Closer to our approach are previous works on hybrid methods, in which the machine learning prediction is refined by knowledge-based reasoning over context data [5], [7]. Indeed, semantic reasoning can exclude from the prediction those activities that are highly unlikely according to the current context (e.g., if the user is at the park, it is highly unlikely that she is brushing teeth). The major issue of those hybrid solutions is the rigidity of the ontological formalism that is based VOLUME 8, 2020 on logical rules that despite offer the power of abstraction, struggle to capture the probabilistic nature of the relationship between context and activities.
There is a vast literature on logic formalisms that support some form of uncertainty reasoning. Some efforts also specifically for applications in the area of pervasive computing [24]. Considering description logics as the underlying logics of ontologies, an integration with fuzzy logic has been proposed to express confidence values for each axiom [25]. ProCAVIAR is not based on fuzzy reasoning but on probabilities, since it associates to each possible activity the probability of occurring in certain context conditions.
One of the well known formalisms that combine logic with probability theory is Markov Logic Network (MLN) [26]. MLN can model both hard and soft constraints using weights associated to each rule. Generally, the weights associated to soft constraints are learned from labelled data. MLNs have been proposed for smart-home activity recognition [27]. However, they are less suitable than ontologies to model the complex hierarchical relationships between context data and activities.
More recently, probabilistic ontologies have been proposed. Examples of such ontologies are PR-OWL [28], [29], DISPONTE [30], [31] and Log-linear Description Logics [32]. Such tools combine the expressiveness of description logics with probabilistic reasoning, and hence they are potentially suitable for ProCAVIAR. PR-OWL relies on OWL2 as a high-level overlay for multi-entity Bayesian networks. Its major drawback is that extending an existing ontology into a PR-OWL one requires significant engineering efforts, since it requires to map ontological concepts to probabilstic entities in the underlying bayesian network. Since re-using and extending existing ontologies is desirable in our domain, PR-OWL is not suitable. DISPONTE extends description logics with probabilistic reasoning using Binary Decision Diagrams (BDDs) to compute the probability of a query based on the explanations generated by the reasoner. However, DISPONTE is based on the closed-world assumption and it considers axioms as independent and identically distributed random variables, which is unrealistic in our domain. Log-linear description logics integrate description logics with probabilistic log-linear models to compute a probability distribution over coherent and consistent worlds. ProCAVIAR relies on log-linear description logics, since it is the most suitable probabilistic modeling tool for our target domain.
To the best of our knowledge, the only application of probabilistic ontologies to activity recognition has been to smart-home activities [33]. In this work, we consider different types of sensors and activities. Moreover, the way ProCAVIAR represents knowledge and exploits the reasoning capabilities of the ontology is also very different.

III. OVERALL ARCHITECTURE OF ProCaviar
In this section, we describe the overall architecture of Pro-CAVIAR, as depicted in Figure 1. Inertial sensors data (e.g., accelerometer, gyroscope, and magnetometer) coming from multiple mobile devices are processed by the Incremental Statistical Model that relies on an incremental semi-supervised classifier to produce a candidate probability distribution over the possible activities. The mobile devices also dynamically acquire context data both exploiting built-in sensors (e.g., GPS, luminosity sensor) and by querying publicly available Web services (e.g., Google APIs to obtain the user's semantic location). Note that, in the literature, ''context'' is a very broad term used to define a situation at different levels of abstractions [8]. For the sake of this work, with context data we denote the information about the environment that surrounds the user (e.g., user's semantic location, temperature, the time of the day) while the user is interacting with the system. Context data are analyzed by the Ontological Probabilistic Model. This module generates confidence values on the possible activities given the current context. These confidence values are then used to refine the statistical prediction. Finally, the Prediction Confidence Evaluation module adopts an active learning strategy based on a dynamic threshold. When the confidence on the refined prediction is lower than this threshold, a query is triggered to the user in order to obtain the ground truth. Upon receiving usable feedback, the module sends a new labeled sample to the incremental statistical model.
More details about the Incremental Statistical Model and the Prediction Confidence Evaluation module can be found in [7]. In the following, we focus on the Ontological Probabilistic Model.

IV. ONTOLOGICAL PROBABILISTIC MODEL
The core of ProCAVIAR is the Ontological Probabilistic Model, implemented through a probabilistic ontology which models relationships between activities and context data. Differently from a standard ontology, it takes into account the intrinsic uncertainty that characterizes these relationships. Context data obtained from the mobile devices are automatically translated into ontological facts, which are then added to the ontology as a description of the current context condition. Then, probabilistic reasoning is in charge of inferring, given the current context situation C, a confidence value conf (C, A i ) for each activity A i ∈ A. Intuitively, conf (C, A i ) estimates the ''semantic compatibility'' of A i being performed by the user whose current context is C. Finally, these confidence values are used to refine the probability distribution h(fv) derived from inertial sensors data.
The context-refined probability distribution is then forwarded to the Prediction Confidence Evaluation module.

1) PROBABILISTIC ONTOLOGY MODELING
Our framework combines soft and hard constraints to model the relationships between activities and context. Hard constraints capture context conditions that should always be satisfied to consider a given activity as possible. For instance, Walking is an activity that requires the user to have a positive speed. On the other hand, soft constraints are useful to capture context conditions that are likely to occur when an activity is performed, but not necessarily they have to be verified; this can be captured by associating a certain degree of confidence to the axiom. Intuitively, the highest the confidence and the more value will have the presence of that context for the likelihood of the corresponding activity to occur. For instance, it is more likely that the activity Running is carried out in a sunny day rather than in a stormy day. Hence, the confidence value associated with the soft constraint ''running can be performed on a sunny day'' should be high, while the one associated with the soft constraint ''running can be performed on a stormy day'' should be lower.
In ProCAVIAR, we modified the publicly available OWL2 ActivO ontology [5] into a probabilistic ontology based on log-linear description logic [34]. A log-linear description logic is characterized by a CBox (i.e., Constraint Box) defined The inclusion of an axiom in C D and C U is mutually exclusive. C D is also assumed to be coherent and consistent (i.e., it is not possible to derive inconsistencies). A log-linear description logic relies on a log-linear probability distribution over the coherent and consistent subsets of the CBox. Each subset of the CBox represents a world that, if coherent and consistent, is associated with a probability computed using the weights of its soft axioms. Incoherent and/or inconsistent subsets of the CBox are considered as impossible. More details about log-linear description logics can be found in [34].  In our probabilistic ontology, activities are explicitly grouped according to context conditions. Examples of these groups could be ''activities that can be performed indoor'' or ''activities that can be performed at a positive speed''. We refer to these groups as activity characterizations. Clearly, an activity may belong to more than one characterization. Characterizations provide an abstraction layer that improves the ontology readability. Moreover, characterizations can be used to define mutually exclusive sets of activities. This approach also makes it possible to easily add new context conditions by creating a new characterization and binding it with the desired activities. Figure 2 and Figure 3 show how characterizations are represented in our ontology.
Each characterization is modeled as an equivalence axiom which describes a specific context condition that an activity should satisfy. Therefore, each activity can be modeled in terms of set membership to specific context conditions by using subsumption or disjunction axioms with the characterization classes.
In order to understand how activities are modeled in our ontology, consider Running. It is clear that a person has to move with a positive speed in order to perform this activity. However, other context conditions related to Running should be modeled considering soft constraints: • outdoor/indoor: even if it is more likely that a person is running outside, it is also possible to run inside a building; • speed: a person may run with different speed rates and each rate has its own probability. Intuitively, a normal running speed rate is the most likely one for this activity, VOLUME 8, 2020 slow running (e.g., jogging) is slightly less probable, while running fast is the least likely one; • height variation rate: a person may run on a flat or inclined road. Hence, users may run at varying height variations. The most likely scenario is probably running on flat roads. Due to these considerations, a possible probabilistic modeling of Running is depicted in Figure 4, using subsumption and disjunction relationships with the corresponding characterizations. The hard rules are recognizable by the absence of the yellow OWLAnnotation marker, which is enabled on the soft rules. Indeed, the specific log-linear logic that we adopted in our system associates a weight to each soft axiom by using an OWL2 annotation called confidence.
Weighted subsumption axioms are used to describe uncertainty about the different values that a context condition can have. As we show in this example, our ontology includes a weighted subsumption for each possible speed rate related to Running.
Note that, for instance, the soft constraint of Figure 4 related to low speed can be formalized as follows: Running Act_Performed_With_SPEED_LOW : w 1 where w 1 ∈ R is the weight associated with this axiom. Later in this section we will discuss how these weights are actually computed. The weight influences the veracity of other axioms related to the same context property (e.g. Running can be performed at medium/high speed rates). Therefore, when modelling weighted axioms, we need to pay attention to how a specific weighted axiom influences the others in the reasoning process.
In our model, weighted disjunctions are used to represent uncertainty about context conditions considered both in hard rules and soft rules. For instance, if we model using a weighted subsumption axiom that Running may be performed indoor, then Running and other outdoor activities would be associated with different output probability values given the same context conditions. This would happen because of the semantics of log-linear DL, which would take into account also the indoor subsumption axioms during the reasoning process of the current user's context, that may specify that the user is outdoor. On the other hand, weighted disjunctions express a degree of incompatibility between an activity and specific context information. In this case, the axiom would be taken into account during the reasoning process only if the current context contains that information. In this example, the weighted disjunction can be formalized as follows: Running Act_Performed_With_LOCATION _Indoor ⊥: w 2 where w 2 ∈ R is the weight associated with the disjunction.

2) AXIOMS' WEIGHTS
In log-linear description logics, the weight associated with a soft axiom takes values in R. In the literature, those weights are generally learned from labeled data. In our domain, the acquisition of a comprehensive annotated dataset that includes activities performed in a wide variety of context conditions is prohibitive. In this work, we associate with each axiom a probability value p ∈ [0, 1] based on common-sense knowledge on context and activities. This knowledge should not necessarily come from the knowledge engineer and domain experts but it may be extracted semi-automatically in several ways, including: • Proposing a survey to a large number of users; • Scraping information about context and activities from the Web. For example, suppose that, according to the commonknowledge, the activity Running is not very likely when performed in indoor environments. Hence, according to common-sense knowledge, our system associates the probability value 0.3 to the soft axiom ''running can be performed indoor'', while 0.7 to the soft axiom ''running can be performed outdoor''.
Note that directly using probability values as weights associated with soft axioms is not a good choice given the underlying log-linear probability distribution. Hence, as proposed in other works [33], [35], we use the logit function to map each probability value p to a real number as follows: The advantage of using logit is that it can approximate probability values for a log-linear model. Note that logit is not defined at 0 and at 1. When p = 1 or p = 0 we consider the axiom as a hard constraint. In the former case, it is a context condition that is always required for the corresponding activity; in the latter case, it is a context condition that should never occur.

3) PROBABILISTIC REASONING
ProCAVIAR uses the previously described probabilistic ontology to compute, given the current context data,  a confidence value for each activity. First, context data is translated into ontological facts: class instances and relationships that populate the assertional part of the ontology. Once the ontology has been extended with facts about the current context conditions, it is processed by the probabilistic reasoner ELOG [34]. ELOG is in charge of computing marginal inference to obtain, for each activity A i , a confidence value. Each confidence value conf (A i , C) estimates the compatibility of A i with the current context condition C.
The marginal inference algorithm implemented in ELOG, called MisSampler (Minimal Inconsistent Subset Sampler), analyzes the entire ontology to generate a posterior probability value for each soft axiom according to the log-linear description logic semantics. In order to compute a posterior probability value for each activity possibly performed by the user, our ontology includes dedicated soft axioms. In particular, there is an additional soft axiom for each activity. Each one of these axioms is declared as subclass of the corresponding activity entity with 0 as default confidence value. According to the log-linear DLs semantics, the posterior probability of these axioms is 0.5 if they do not conflict with other axioms. Indeed, without conflicts, the posterior probability of an axiom c with weight w c is defined by alogit(w c ) where alogit(R) → (0, 1) is the logit inverse function. Figures 5 and 6 show those additional soft axioms and their relationships with the rest of the ontology.
The output of the marginal inference algorithm is a vector of confidence values: confidences(C) = c 1 , c 2 , . . . , c n where C is the input context data and c i ∈ R+ is the confidence value conf (A i , C) associated to the activity A i ∈ A. Note that conf (C) is not a probability distribution over the activities. Each c i is a posterior probability value computed by the underlying log-linear probability distribution over coherent and consistent ontologies. Hence, these values should be considered as confidence values associated with the activities given the current context condition. Since we use the default value 0 for the probabilistic axioms in the ontology, the value of each c i is in the range [0, 0.5] (due to alogit as we previously discussed).
Given a confidence value c i : • c i = 0.5 reveals that the context satisfies the hard rules related to A i without the involvement of probabilistic axioms; • 0 < c i < 0.5 reveals that the context satisfies the hard rules of A i but some soft axioms were used in the inference process, thus decreasing the output confidence. The more soft axioms are involved, the lower the output confidence; • c i = 0 reveals that the context does not satisfy at least one hard constraint for A i . Therefore, according to our ontology, the activity is impossible in that specific scenario.

4) REFINED ACTIVITY PREDICTION
The confidence values inferred by ELOG are used to refine the probability distribution obtained from the Incremental Statistical Model on inertial sensor data. In particular, given the probability distribution p 1 , p 2 , . . . , p n and confidences(C) = c 1 , c 2 , . . . , c n such that A i is an activity label, p i is the probability associated to A i by the statistical model and c i is the ontological confidence value of A i given the current context condition, we compute the following vector: v = p 1 * c 1 , p 2 * c 2 , . . . , p n * c n Hence, confidence values are used as weights associated to the probability values. Finally, the vector v is normalized in order to obtain a probability distribution over the possible activities: predictions = P 1 , P 2 , . . . , P n such that n i=1 P i = 1 and P i ∈ [0, 1]. This probability distribution is the output of the Ontological Probabilistic Model and is forwarded to the Prediction Confidence Evaluation module.

V. EXPERIMENTAL EVALUATION A. DATASET
In order to evaluate the effectiveness of ProCAVIAR, we used the dataset collected in [7]. To the best of our knowledge, there is no other dataset with rich contextual information that can be exploited by ProCAVIAR. The considered activities are the following: walking, running, standing still, lying, sitting still, stairs up, stairs down, elevator up, elevator down, cycling, moving by car, sitting on transport, standing on transport and brushing teeth. Those activities were executed by 26 volunteers in different context conditions, including working at the office, going around in the city (Milan), driving, using public transportation, cycling, and staying at home. For each user, the dataset includes inertial sensor data and context data collected from a smartphone in the front pocket and a smartwatch on the wrist. Overall, the dataset includes 9 hours of recorded activity data (∼ 350 activity instances). More details about this dataset can be found in [7].

B. SIMULATING UNUSUAL SCENARIOS
The dataset that we use to evaluate our system does not include activities executed in context conditions that are unlikely but not impossible in realistic scenarios. For instance, the Running activity was never executed in indoor environments and/or with lower speed rates (e.g., jogging). Another example is the Stairs up and Stairs Down activities, that were never executed outdoor despite it is entirely possible to find stairs outside. Consider a rigid knowledge-based approach to model context data, like the one proposed in [7], where most of the uncommon context conditions for an activity are considered as impossible. Using such a rigid approach in realistic scenarios would negatively impact on the recognition rate. Since we want to quantitatively show that our probabilistic reasoning framework overcomes these drawbacks, we slightly modified the dataset in order to incorporate unusual context scenarios. To the best of our knowledge, there is no public dataset where activities are performed in a wide variety of usual and unusual context conditions.
We implemented a probabilistic simulator for context data which is based on the considered dataset. Hence, we replaced the original context data with simulated context data. For each activity class in the dataset, our simulator considers: • context information which characterizes the activity regardless of the scenario (i.e., context data needed to satisfy the hard constraints of the ontology); • a probability distribution over the context data that may be relevant for estimating the probability for the activity (i.e., context data captured by soft constraints in the ontology).
Our simulator relies on a probabilistic representation of the common knowledge of the activity domain to generate possible scenarios for each activity class. For each activity instance in the dataset, the simulator generates, based on the label, a scenario that includes context data related to hard constraints and some of the context data related to soft constraints. The latter are sampled from a probability distribution.
For instance, consider the activity Walking. Based on common-sense, our simulator incorporates the following probabilistic knowledge: 1) it is very common that users walk slowly (80% of probability), while they sometimes walk faster (20% of probability); 2) in the majority of the cases, users walk on flat surfaces, hence with no height variation (70% of probability), while they can walk ascending/descending paths with a lower probability (30% of probability); 3) Walking can be performed indoor or outdoor with equal probability.
For each activity instance, our simulator generates context data by sampling from these probability distributions. Continuing the example of Walking, a wide variety of context scenarios can be generated, like the following ones: Intuitively, scenario A is the most common one for Walking and it would be frequently generated by our simulator.
The other examples of scenarios are least common, so they would be rarely generated by the simulator.

C. RESULTS
In the following, we present the results of ProCAVIAR evaluated on the dataset presented in Section V-A with context data simulated as described in Section V-B. The probability values associated to soft axioms in the ontology used for the experiments have been chosen as a result of an internal small survey. In order to evaluate the effectiveness of our technique with respect to alternative approaches, we considered two additional methods. The former does not rely on any kind of semantic refinement procedure and it is called Data-driven approach: it combines the incremental activity recognition module and the prediction confidence evaluation module without applying context-refinement. Note that Data-driven approach can be considered as a solid baseline, since it is the standard approach in the literature for activity recognition on mobile devices [1]. The latter is the CAVIAR method [7], that uses a state-of-the-art context-refinement procedure based on standard (deterministic) ontologies. This approach does not take into account the intrinsic uncertainty and incompleteness of common knowledge, and it excludes from the prediction based on inertial sensors those activities that are very unlikely when considering the current context conditions. To the best of our knowledge, there are no other works in the literature that use context data to achieve the same goal. In order to guarantee a fair comparison, in our evaluation we consistently used the same hyper-parameters and classifier proposed in [7]. We performed leave-one-subject-out cross validation to evaluate the recognition rate of our system and the ones of the baselines. Table 1 shows the results in terms of overall F1 score).
Our results confirm that context data has a significant impact on the overall recognition rate. Most importantly, the probabilistic context refinement implemented in ProCAVIAR significantly outperforms the baseline methods reaching an overall F1 score of 0.92. Thanks to its probabilistic perspective, our method can recognize activities performed in unusual scenarios, considered as impossible by the deterministic reasoning approach. Looking closely at the results, some of the activities related to the highest improvements are Going downstairs, Going upstairs, Walking, SittingTransport, StandingTransport and BrushingTeeth. For these activities, our simulator generated a wide range of unusual scenarios, thanks to a higher number of combinations of context data with respect to other activities. Thus, the dataset contains more ''unusual samples'' for those activities respect to the others, which are characterized by a smaller range of possible scenarios.
Considering the solution based on deterministic reasoning, it is possible to observe a significant decrease in the recognition rate of Running. Indeed, while this activity can be reliably recognized only by analyzing inertial sensors data, the deterministic semantic refinement often considers it inconsistent considering unusual scenarios. For instance, the deterministic ontology in [7] considers as impossible the fact that Running can be carried out indoor, since it is unlikely. Our method can overcome these problems thanks to soft axioms.
Besides the recognition rate, a crucial evaluation parameter is the number of questions triggered by the system, since it has a significant impact on usability. Figure 7 shows how both ProCAVIAR and Deterministic reasoning generates a significantly lower number of questions (respectively 6% and 8%) compared to No context (22%).
ProCAVIAR slightly decreases the number of user questions with respect to Deterministic reasoning. Hence, on average, our probabilistic context refinement method further reduces the uncertainty on the output probability distributions compared to the deterministic solution.
In order to evaluate how the recognition rate and the number of triggered questions evolve over time, we use the evaluation method proposed in [36]. We classify each data  sample of the dataset (considering all 26 subjects) with the current model and, depending on the prediction's confidence, we update the recognition model. The classification's output (i.e., the most likely activity), and the corresponding ground truth are collected in sliding windows of 800 samples with an overlap of 75% to periodically compute the overall F-1 score and the percentage of triggered questions. Samples coming from different users are randomly interleaved. Figure 8 shows the evolution of the F-1 score and the number of questions of ProCAVIAR with respect to the baselines. Compared to No Context, both Deterministic Reasoning and ProCAVIAR quickly reach high recognition rates and a significantly lower number of questions. ProCAVIAR significantly outperforms Deterministic Reasoning, showing a faster learning curve. On the other hand, the number of questions generated by Pro-CAVIAR is only slightly lower than the ones generated by the deterministic approach, thus reflecting the results presented in Figure 7.
As we mentioned in Section V-A, to the best of our knowledge there are no other datasets with rich context data that can be used to directly evaluate our context-aware method. Nonetheless, we decided to perform additional experiments on a well known public benchmark for activity recognition: the PAMAP2 dataset [37]. This dataset only includes inertial sensors data gathered from multiple mobile devices. The set of target activities is quite rich and includes the following: lying still, sitting still, standing still, walking,   [38]. Simlarly to our previous experiments, we performed a leave-one-subject-out crossvalidation. Table 2 shows that the machine learning module of ProCAVIAR outperforms the baseline confirming that we build our refinement on competitive machine learning techniques.
While we could not quantitatively evaluate the effectiveness of the refinement obtainable by the Ontological Probabilistic Model on the PAMAP2 dataset for the lack of context data, in the following we explain how additional context information could be used by ProCAVIAR to further improve the recognition rate. We first analyzed the confusion matrix of our Data-driven approach on the PAMAP2 dataset depicted in Figure 9.
A first observation concerns the activities Going Upstairs and Going Downstairs that have a lower recognition rate than others, and are confused among themselves. Since these activities are also part of our dataset, the reader can see in Table 1 the improvement that could be achieved by probabilistic ontological reasoning. By looking at the confusion matrix, we also note that the activities Sitting still and Lying still are often confused by the classifier. Since Lying still is mostly performed in those indoor environments that have beds or sofas (e.g., home environment, hotel rooms, etc.), by considering this knowledge and the semantic location of the user, an ontology-based refinement would be able to improve the recognition rate by excluding that activity as incompatible when the semantic location is some outdoor space. However, even if much less likely, this activity can also be performed in other scenarios, for instance in a park or on the beach on a sunny day (i.e., a weather condition that favors leisure time outside). Hence, ProCAVIAR would be more flexible, defining probabilistic knowledge about semantic locations, time (e.g., it is more likely to lie during the night while sleeping) and weather conditions (e.g., it is less likely to be lying in the park during a rainy day). Similarly, this type of probabilistic reasoning can be applied to other activities that appear to be confused from the matrix, like Running and Rope jumping; Ironing and Standing still. There are clear limitations to this approach when activities can be performed in very similar context conditions, like it may be the case for Standing still and Sitting still.

VI. CONCLUSION AND FUTURE WORK
In this paper we presented ProCAVIAR, a hybrid real-time activity recognition framework based on semi-supervised learning and probabilistic knowledge-based reasoning. Our method applies machine learning algorithms on inertial sensors data to obtain a candidate probability distribution over the activities possibly carried out by the users. Then, a probabilistic ontology that captures probabilistic relationships between context data and activities is in charge of refining the candidate prediction using available context data (e.g., the semantic location of the user, its speed, weather conditions, etc.). Thanks to active learning, ProCAVIAR can continuously improve the semi-supervised classifier, which is initialized only with a limited set of labeled examples.
Differently from existing solutions based on a rigid formalism, ProCAVIAR takes advantage from probabilistic reasoning to capture the intrinsic uncertainty of context modeling. A preliminary evaluation shows that ProCAVIAR actually mitigates the problems of hybrid ontology based solutions while increasing the advantage of hybrid solutions over purely statistical approaches.
This work only represents a preliminary investigation of the effectiveness of using probabilistic logics in context-aware and hybrid activity recognition systems. We foresee several promising research directions.
First, a critical aspect is the setting of weights for the soft axioms determining the influence of context on activities. We plan to investigate how to populate the probabilistic ontology in a semi-automatic fashion, by extracting knowledge about context and activities from external sources. For instance, some works proposed to extract information from textual description [39] and images [40] of activities from the Web. Those works were mainly focused on building models for smart-home activity recognition.
Beside uncertainty on the association of context with activities, a probabilistic ontology may also capture the fact that context data may have an associated confidence value. Indeed, it is not always advisable to completely trust input context data (e.g., geographical positioning, as well as semantic place identification, can have different levels of approximation and reliability). Including uncertainty on input data has the potential of making our system more robust with respect to inaccurate information.
Finally, we aim to study personalization aspects. Each user may have her personal habits, and hence personal context situations. Incrementally adapting the probabilistic ontology to each user would allow our system to learn personalized contexts and hence improving the accuracy and scalability of our approach.