A context-aware model for human activity prediction and risk inference in actions

Even though human activities may result in injuries, there is not much discussion in the academy of how ubiquitous computing could assess such risks. So, this paper proposes a model for the Activity Manager layer of the Activity Project, which aims to predict and infer risks in activities. The model uses the Activity Theory for the composition and prediction of activities. It also infers the risk in actions based on changes in the user’s physiological context caused by the actions, and such changes are modeled according to the Hyperspace Analogue to Context model. Tests were conducted and the developed models outperformed proposals found for action prediction, with an accuracy of 78.69%, as well as for risk situation detection, with an accuracy of 98.94%, showing the efficiency of the proposed solution.


Introduction
Low cost and reduced size sensors are more available nowadays.This and the fact that mobile device technologies are well established, allowed for advances in research on the area of ubiquitous and pervasive computing (d'Auriol et al., 2007).These mobile devices have embedded sensors that gather different types of data which can be used for the representation of the user's state as well as the environment surrounding him or her.Microelectronics is advancing and contributing to research that aims to create smart environments, such as smart houses, health care systems and service recommendation systems.So, in order to provide customized services, such as the ones previously cited, it is necessary to understand how a user interacts with the environment.That is, the systems need to have access to data related to the context surrounding their users and, especially, analyze the interaction between a user and the environment based on the detection of activities that are performed by the user (Mocanu and Florea, 2011).Therefore, we can consider the detection of activities as a crucial point in smart environments, because they are a part of the user's context (Lu and Fu, 2009;Crispim et al., 2012).In order to expand the knowledge about the user's context, we consider that the detection of risk situations related to an activity has an equally important role, due to the fact that it allows the collection of contextual information about the situation of the user while he or she performs an activity.The recognition of risky activities helps determining the quality of the interaction between the user and the environment.Furthermore, such recognition improves the contextual information in such a way that a cycle is established, where each activity performed by the user improves the understanding of the personal context and its relationship with the environment (Mocanu and Florea, 2011).
Therefore, this work presents a model for inferring risk situations based on the user's actions, which are a part of some activity.When considering the Activity Theory (AT) (Vygotsky, 1979) and the Hyperspace Analogue to Context contextual model (HAC) (Rasch, 2013), the proposed model is able to infer the probability of the execution of the next action to be performed through the analysis of the user's normal behavior, and the detection of risk situations is made by monitoring the user's physiological data while he or she performs some activity.In this way, we can associate a risk situation to each action or activity that will be performed based on the user's normal behavior.This work is part of a larger project for the detection of risk situations called Activity Project.It consists of a context-aware mid-dleware (Figure 1) that encompasses the proposed model in its Activity Manager layer (Del Fabro Neto et al., 2013).
This paper is structured as follows: section Background presents the background needed for the understanding of the model proposed in the work; Proposed model describes the proposed model and its functioning; Evaluation discusses the evaluation of the action prediction and the detection of risk situations in the actions; Related works presents the works related to action prediction and risk detection; and Conclusion contains our final considerations and refers to future works.

Background Activity Theory (AT)
When considering the AT, an activity is said to be composed of actions that are considered atomic units, that is, they cannot be divided into more actions, and the basic idea behind the AT is that the subject is participating in an activity because he or she wants to achieve some specific goal (Vygotsky, 1979).The object of an activity is the focus of the subject's interest, where he or she wishes to use and/or modify it in order to achieve some expected result.The interaction between the subject and the object is mediated by artifacts.Thus, a basic triangle between subject, object and the mediation artifacts is formed (Figure 2).Even though it is possible for systems based on the basic model of the AT to recognize activities, this basic model of the AT does not encompass all the components that are relevant for the recognition.One should also take into consideration, in the mediation of the subject with the com-   (Kuutti, 1996).munity, the fact that the subject also has social and cultural contexts.
Because of this, the Cultural-Historical Activity Theory (CHAT) (Figure 2) inserts the community component that is mediated by rules and division of labor (Kuutti, 1996).These concepts are hard to be represented computationally, so the context taxonomy (Figure 3) was developed to provide a pragmatic view in the construction of artifacts, and it also incorporates general concepts of the AT into context-aware systems (Mikalsen and Kofod-Petersen, 2004).Thus, the CHAT components can be compared to the taxonomy of contextual knowledge as presented in Table 1.

Composition of activities
In this paper we use the AT for the contextual modelling of an activity.However, it is necessary to define the structure of an activity, since the manner in which actions are related to each other and to the activity they are part of is not defined from the computational point of view when considering the AT.In this way, we use a hierarchical model of activities for the activities of daily living (ADL), which can be extended to any system that defines an activity as a composition of actions (Naeem et al., 2007).ADLs are composed of actions or other ADLs of a lower level.For their representation, a plan-representing, task-specific and intention-oriented language, called Asbru, is used.In this language, an ADL can be classified as mandatory or optional.When an ADL has subgoals (or sub-activities, or action) classified as mandatory, they must be performed before the ADL is said to be executed.If they are optional, the activity may be recognized without the execution of all actions that compose it.
Actions can also have a defined order of execution.For example, they can be performed in a sequential manner with a strict order of execution, in parallel when they are performed at the same time, in any order with only one action being performed at a time, or they can be defined as not having an order of execution, which means that the activity does not require a synchronization between actions.In this way, each activity can have a model in the system based on these concepts for its detection.Such a model also allows the addition of temporal restrictions, making it possible to use some time window for the detection of activities.The temporal restriction can be applied to the ADL or the actions that it is composed of.

Hyperspace Analogue to Context (HAC)
Context attributes are important pieces of information needed for the detection of human activities.In order to structure them, we adopted the HAC model (Rasch, 2013).This model characterizes the context in multiple dimensions with values that can vary according to some predefined thresholds.With this, it is possible to take some action when one detects that some context attribute has a value outside Table 1.Basic aspects of an activity and their relation to a taxonomy of contextual knowledge (Kofod-Petersen and Cassens, 2006).Source: Mikalsen and Kofod-Petersen (2004).

Subject
its threshold.Besides, the HAC model has a well-defined syntax and defines operations that allow dealing with contextual data.However, the main advantage about this model is the possibility of capturing every context change, which is an important piece of information for understanding the user's behavior.In this way, the environment's and user's historical contextual information can be checked for the inference of risks in actions that make up the activity being developed at the moment as well as the risks in the possible inferred future actions.Thus, if a user performs an action that yields a specific context change, it is possible to assess whether such a change would yield a risk situation.
Performing simulations of context changes over the current user's context permits unknown scenarios to be checked with the intention of probabilistically predicting whether a certain action will yield a risk situation from the generated context.In this work, we use context changes to infer whether the change in a certain context attribute (or dimension) can result in a future context change that may cause some harm to the user, that is, if it would drive him or her to a risk situation.

Proposed model
This work defines the Activity Manager layer for the Activity Project (Figure 1).The proposed model has different purposes: (i) the detection of actions and activities; (ii) the assignment of a risk situation to each action; (iii) and the inference of future actions and activities for the prediction of risk situations.The way in which the model is structured is presented in Figure 4 and its flow is as follows: after receiving the aggregated sensor data from the Sensor State layer of the middleware, the first step is to recognize the action being performed and infer the next action to be performed, based on the history of actions and activities executed by the user.This is made in order to analyze risks in future actions to be performed by the user, so that the inference of risks can be done in future contexts for the user.The process to recognize the action being performed is made by the Action Recognition module presented in Figure 4, and the process to infer the next action is realized by the Next Action Inference module also presented in Figure 4.
In possession of the next action to be performed, two different processes are initiated in parallel: (i) the Activity Prediction and (ii) the Action Risk Inference.
The activity prediction (i) in this model is made by the Activity Prediction Component, which receives the probable next action, assumes it will actually happen and requests a new future action to the Next Action Inference component.This is made by using as the current action, the one previously inferred as the next action to be performed.In this way, the Activity Prediction Component repeats this process until a sequence of actions that represent an activity is recognized.This sequence of actions is structured in accordance with the Asbru Language (http://www.openclinical.org/gmm_asbru.html)through the Asbru Hierarchical Model.
For clarification purposes, one can imagine that the Action Recognition component recognized the current action a 1 , and the Next Action Inference component detects that the probable next action is a 2 .The Activity Prediction Component requests from the Next Action Inference component the probable next action using the action a 2 as its basis.The process involved in making this inference is explained in Activity recognition and activity prediction; besides that, the Action Risk Inference module associates a risk situation to each predicted action based on the context changes and on the future contexts yielded by these changes.Each of these components is explained in the next subsections.

Action recognition
The aggregated data from sensors is transmitted to the Action Recognition module and compared to the already classified data for each action registered in the system based on classification algorithms.This way, the set of sensor data will be classified according to the similarity that it has with the registered data for each action that the system is capable of recognizing.This is made using machine learning algorithms.
Six machine learning algorithms for classification based on raw sensor data for the recognition of actions are used (Dernbach et al., 2012).They are: Multilayer Perceptron, Naive Bayes, Bayesian network, Decision Table , Best-First Tree and K-star.As stated by Dernbach et al. (2012), in order to obtain better results with these algorithms it is necessary to apply the Feature Extraction process in raw data.We apply the feature extraction process in the time domain because of its performance and implementation simplicity in comparison with methods such as Fourier Transform.The time domain feature extraction methods chosen in accordance with Lara and Labrador (2013) were mean, standard deviation, variance, IRQ (interquartile range), RMS (Root Mean Square), MAD (Mean Absolute Deviation) and correlation between pairs of axes.
In this work, we used the WEKA software in order to apply machine learning algorithms to the data, with default patterns associated with each of the classifiers and applied to the data after the feature extraction process.About 30% of the dataset was used as training data.As a result of the tests, the best algorithm, that is, the algorithm that was able to correctly classify most of the samples, was the K-star, and so it was the one chosen to be used in our model.

Next action inference
We base the inference of the next action on the user's historical data on already performed actions so that we can reflect his or her usual behavior.Figure 5 depicts the algorithm developed.It is based on the search for patterns of activities (or actions) in the user's history H, that is, the search for certain sequences of activities with the objective of discovering which action is the next one to probably be performed according to these patterns.
Briefly, the algorithm presented in Figure 5 searches the list A of the last MPL performed actions, where MPL is the window size of the quantity of actions to be analyzed.Afterwards, a search is made in the list of patterns p in the user's history for the previously found actions in A. Thus, the number of occurrences for each of the actions a is updated for each pattern p found and related with the maximum of MPL periods with each action a.The highest occurrence found is the probable future action.

Activity recognition and activity prediction
In this work, we assume that activities are composed of actions and that the relationship between such actions tells how an activity happens.To achieve this goal we use the Asbru Hierarchical Model together with a simi- lar approach presented by Naeem et al. (2007), which allows us to define whether actions are or not mandatory and whether they must be performed in a defined order.In this way, an activity is detected when a certain set of actions related to the activity is performed in some established time window.
In our adaptation of these models, two types of actions are defined: (i) core actions and (ii) secondary actions.The former represent actions that are essential for the person to achieve the goal of the activity, so that they are actions that need to happen for the recognition of the activity.The latter are actions that can happen during the performance of an activity but are not essential for their recognition; they are useful for adding meaning to the activity.For an activity to be recognized, each action that it is composed of must be executed within a predefined time window.In this way, an initial time window must be determined, and during the preliminary phase it is adjusted by the system.Since this window depends on the frequency of each activity, it varies according to the user's behavior.Figure 6 presents an example of activity recognition.
In the example, there are three activities that the system is able to recognize: X, composed by the actions a 1 and a 5 , with a time window of 3 periods; Y, composed by the actions a 2 , a 3 and a 4 , with a time window of 5 periods; and Z, composed by the actions a 1 and a 4 , with a time window of 2 periods.The actions detected in a certain period t i are arranged inside a box.For example, in period t 4 the detected actions are a 1 and a 7 , because the time window only includes the last detected actions.In this way, it is possible to note that activity X was detected, since during its time window the actions that it is composed of were performed and detected (a 1 in t 4 and a 5 in t 5 ).The same applies to activity Y, but not to activity Z.
When an action that is part of some activity is detected during the time window for that activity, the latter is started.It only ends when its last pending action is detected.For the activity to be complete, all of its core actions must be detected during the time window; if they are not detected, this means that only some of its parts were performed.This is the case of activity Z in figure 6.In the example, X, Y and Z are happening simultaneously, although activity Z is not complete.
In this way, we can predict an activity using the probable future actions.So, if the future actions fit the model defined for an activity in particular, we can predict the activity that will probably be performed.

Action risk inference
When inferring risk situations, we must account for the fact that each different user has his or her own behavioral pattern.This is so because people are different and are considered beings of habits (da Rocha et al., 2010).To determine a risk, the changes that an activity yields for the context have to be analyzed.This is done by the Search Context Changes module, which searches in historical data what were the context changes yielded by the predicted action.It is worth mentioning that this analysis has to occur before an action is actually executed by the user.That is, the system must predict actions and, consequently, activities that the user might perform and, thus, identify whether these future actions or activities will drive him or her to a risk situation.This approach implies the need for (i) capturing the current user context, (ii) inferring which is going to be the next action(s) to be performed, (iii) applying the context changes resulting from this action in the current context (Current Context Capture) and (iv) evaluating the resulting context while looking for risk situations.
Our solution for this problem (prediction of actions) is composed of a (i) component to discover context changes based on historical changes in context that were caused by each action (Context Change component); and a (ii) component to simulate future contexts (Future Context Generation in Future Context Viewer) resulting from the changes (Action Risk Inference subsystem, Figure 4) found.Based on the predicted action, the Context Change component searches in the user's historical data which context changes were previously caused by it.Then, the context changes found are applied in the current user context by the Future Contexts Viewer, so that a new context is generated, which represents the future state of the current context if the predicted action is actually performed.After finding out this future context, the Future Contexts Viewer checks the user's profile through the Search User Profile component to discover if the new context is not outside the safety thresholds pre-defined for the user (Risk Situation Generation).If it is outside, the user is considered to be in a risk situation.These thresholds (represented in the User Profile) are updated for each action that the user performs; in this way, they represent the user's capabilities to execute each action.

Evaluation
The detection of risk situations proposed in this work is based on the prediction of actions performed by users' previous behavior, as well as on context changes yielded by such action.In this sense, it is necessary that the system is (i) able to correctly predict the probable next actions to be executed, as well as (ii) estimate whether the context changes yielded by an action will imply a risky context for the user.In order to validate the proposal of this work, we conducted two different experiments.The first one intends to analyze the accuracy of the presented prediction model, while the second one intends to validate the model for the detection of risk situations based on the context changes yielded by actions in the user's context.

Evaluation of action prediction
In order to evaluate the proposed model in this work, we chose to use a public dataset called Aruba Dataset (Cook, 2010), because it allows one to compare the results of the model of actions predictions with other correlated proposals, since it is a widely used dataset in research of activity recognition in the ubiquitous computing area.The dataset has 11 different activities registered using 42 sensors.Thus, in this work, the accuracy obtained from the dataset used was 78.69%.

Evaluation of the risk situation in actions
To perform the evaluation of risk in actions, we had to create a dataset with annotated actions and with physiological data, because we did not find a public dataset with such information.The created dataset is composed by the actions of lying, walking, running, sitting and standing.These actions were captured from gyroscope and accelerometer data coupled in a smartphone.Heart rate was the physiological data gathered from a sensor connected in an Arduino.
In this way, from the 2455 entries of the dataset, 49 risk situations were detected by the model, achieving 98.94% of accuracy.The obtained accuracy was measured based on the values false-positives (13), false-negatives (13), true-positives (36) and true-negatives ( 2393).These values were generated by the analysis of the comparison between the predicted thresholds for the current action, the current heart rate and the real thresholds for the current action.The real thresholds are obtained from the average of the historical values of the heart rate for the current action.However, we use three times the standard deviation for these values because thresholds that consider the average of the values subtracted and add up to three times the standard deviation represents almost all values in a normal distribution.The determination of the values of the predicted thresholds are based on the preceding action, where the median of the context changes performed by this action and also the value of the heart rate while it was being were used.
To perform the activity prediction test we use the Aruba dataset, considering recognized activities as actions.In this way, we can create the activity Have Meal composed by a group of actions present in the Aruba dataset.The activity Have Meal consists in the act of preparing meals for later consumption.As explained in Proposed model, the process for specifying an activity consists in three different phases: (i) specify the actions that the activity is composed of; (ii) specify whether the actions are core or secondary actions; (iii) specify whether the actions have or not an execution order.The activity Have Meal is composed by the core actions Meal Preparation and Eating, and by the secondary actions Wash Dishes and Relax.Furthermore, we define that this activity has an execution order, with the Meal Preparation action always happening before the other actions.However, these other actions do not have a defined order for execution.This was set with the purpose of avoiding cases where, for example, the user is eating a meal that does not require its preparation.
Applying the algorithm for activity prediction, we obtained an accuracy of 70.69%.The activity prediction algorithm uses the action prediction algorithm presented in Figure 6 to infer the future actions that will occur after the current action in order to also infer an activity within a predefined time window.By analyzing the distribution of the dataset, one can notice that the greatest part of this accuracy is related to how the activity was modeled.In the dataset used, it was possible to identify the user's behavior in the sequential execution of the action Eating after the action Meal Preparation, both within the time window.Since the action Meal Preparation is always the first one to occur, according to the predefined order for the Have Meal activity, the detection always occurs after this first action is executed.Curone et al. (2010) present a system for classification of emergency situations focused on risky jobs like those of firemen and Civil Protection rescuers.The operators are equipped with two sensors (accelerometer and ECG sensor), through which the system is capable of recognizing many user states that correspond to many ADLs in real time, taking into consideration the information of the sensors used.The system had 88.8% of accuracy in the activities of classification with the tests performed in a laboratory.

Related works
The work of Wang et al. (2014) presents a distributed approach that employs the computing and storage resources in each node of a wireless sensors network (WSN) to detect abnormal activities.A normal activity is defined as the combination of duration and trajectory, and an abnormal activity is defined by the authors as an activity which has a trajectory and a duration that are significantly different from a normal activity.To determine the normal behavior of a user, the authors performed a frequent pattern mining in order to find the patterns of normal activities considering their trajectory and duration.In this case, if the frequency of an itemset exceeds the minimum threshold defined in the system, it is classified as a normal activity.By performing some tests in an environment simulated by software, the accuracy was 96.2%.
Gil-Quijano and Sabouret (2010) presented a proposal for the prediction of household activities in a smart home.The goal of the work is to adapt the behavior of the house applications on the basis of the predicted human activities by correcting the behavior of devices and prepare the rooms to receive people in a condition pleasant for them.The proposal for the activity prediction is based on the construction of a directed graph for each occupant from the statistical analysis of the activities performed by him or her.The graph nodes represent the tasks and the edges represent the sequence of execution between two tasks, where they have the probability of execution of their sequence assigned to themselves.Thus, each task is performed in a given environment and permits the prediction of the next displacement in the graph based on the current task, featuring the prediction of activities.
This paper also uses an approach for prediction based on the analysis of frequent patterns found in the user's history in order to determine the next action.The main difference of the proposal presented in this paper is the algorithm used, which considers a variable window for the search of frequent patterns for each executed action.In this way, the algorithm searches in the user's history all patterns that happened within a predefined window and determines the next action based on the pattern with the highest number of occurrences.Mocanu and Florea (2011) developed a model for the recognition of ADLs and detection of emergencies.The emergency detection is performed by a context-free grammar with the aid of ontologies for the modeling of activities.The authors consider four types of emergencies and classify them as: duration emergency, for activities that last longer than usual; context emergency, for activities that are performed in a place considered to be wrong; unusual time emergency, for activities that are performed in a time that diverges from the usual; non-natural order emergency, for activities that are performed in an order that is not normal for the subject being monitored.
In order to detect if an emergency occurs, four attributes (duration, place, time of day, activities order) are compared with a daily program that each user has.This program describes all the activities that the user normally performs.Any difference between what is being performed and what is in the daily program generates an emergency.The authors used a simulated environment for the activities of reading, watching TV and eating and simulated the emergencies of context, duration and time of day.The metrics used for the detection of emergencies were accuracy, with a value of 91.75%.

Comparison
To compare our proposal with the related works presented in the previous section we considered some aspects, such as: accuracy of the approach, dataset utilized, category (action pre-diction or risk detection), algorithm employed and the attributes used for risk detection.Tables 2 and 3 summarize this comparison.
In the work of Curone et al. (2010) the detection of risk situations is performed using a pre-determined set of combinations between the known activities, like a rule system.For the system recognize new risk situations, new possible combinations between the activities must be added.In our work, the variation of the user's physiological data while he or she is performing an activity is used to adjust the thresholds based on the user's history for the risk situation detection.Thus, the system can use the various ways the user performs an activity to adapt itself and is more flexible.
The approach presented by Wang et al. (2014) is similar to ours, where their system considers risk situations as deviations in the user's normal behavior.However, in their work they do not consider the changes in the physiological aspects of the user, but only the duration and the trajectory of the activity's execution.In this case, the system does not detect risk situations where the physiological aspects are outside the thresholds of the user.Curone et al. (2010), Mocanu andFlorea (2011), andWang et al. (2014)   ity prediction to infer risk situations, while we use activity prediction (as a set of actions that an activity is composed of) to infer these situations in the future based on the probability of the execution of each action.The works presented in Table 2 use private datasets and different attributes to infer risk situations, so we present the results obtained in each work using their own dataset.The algorithm used in each work and the accuracy obtained are also summarized.Gil-Quijano and Sabouret (2010) proposed an approach based on the use of a directed graph for action prediction, considering the ratio between how many times a sequence of two actions was performed by the person and the number of times that the initial action was performed in the same edge.However, we had better results in comparison with this approach because in our proposal the system can search for an appropriate pattern length for each case using the user's historical data, not only considering the last action, but also taking into account a higher number of previously performed actions.
In this way, considering algorithm 6, the best value for MPL that we achieve is 2, getting an accuracy of 78.69%.This result is better than the 61.28% of accuracy obtained by Gil-Quijano and Sabouret (2010) using their directed graph algorithm, which represents a gain of 28.41% in the future actions inference.Both works are summarized in Table 3 using the same dataset for comparison.

Conclusion
To assist users in decision-making, the context-aware systems need to act in a preventive manner, taking into consideration the prediction of risk situations.In this sense, this work presented a model for action prediction and detection of future risk situations based on the Hyperspace Analogue to Context (HAC) and on the Activity Theory.In the technique of action prediction we obtained an accuracy of 78.69% and in the evaluation of risk situations an accuracy of 98.94%, thus achieving better results than the related works.For future work we intend to improve the approach for action predictions, taking into consideration the performance of algorithms.We also intend to realize tests with public datasets with a greater amount of physiological information to perform a complete evaluation of the detected risks.

Figure 1 .
Figure 1.Architecture of the activity project.

Figure 4 .
Figure 4. Proposed model for the Activity Manager layer.

Figure 6 .
Figure 6.Example of the method used for the activity recognition based on the AT.

Table 2 .
do not use activ-Comparison between related works and the present work using our own dataset.

Table 3 .
Comparison between related works and the present work using Aruba dataset.