ActiviTree: Interactive Visual Exploration of Sequences in Event-Based Data Using Graph Similarity

2.1 Visualization of event sequences

There are several examples of identification of event patterns using visual representations.

Torsten Hägerstrand [11] formulated a conceptual framework called time geography for describing human behaviour in time and space. In time geography an individual's movement is represented by a continuous vertical trajectory, the 'space-time path', within a three dimensional structure, the 'space-time cube', representing space in the horizontal plane and time in the vertical dimension. Several individuals' trajectories can be represented in the same cube and 'bundles' of trajectories can be identified indicating meetings of individuals and revealing patterns of spatio-temporal movement. Kraak in [14] implemented the space-time cube in an interactive geovisualization environment. Kwan has made substantial use of time geographical representations for exploring human activity patterns [15]. The space-time cube has also been used to represent and analyse other types of spatio-temporal data, for example paths of clouds [21] and discrete earthquake events [9]. Even though time geography is an intuitive and effective way of conceptualizing continuous change in time and identifying patterns, it quickly gets cluttered with increasing data size.

The time geographical framework was extended in [6], [7] to consider performed activities in time and the original geographical space was replaced with an abstract activity space. In this approach a continuous trajectory, the 'activity-time path', is used to represent the activities performed by an individual during a period of time. The activity-time paths are drawn within a simplified time-cube representation. Patterns of activities are, in this case, defined as sequences of activity events and are identified within such a representation by being highlighted in the activity space. [7] The work in [27] also focuses on representing and identifying patterns of performed activities using several representations. A 3D rod representation over a geographical map and 2D and 3D activity ringmaps are used, to show activity trends during the day. Another early example of finding patterns of events, in this case personal histories, using visual inspection is Lifelines, presented in [18]. In this work several aspects of a single history are listed as vertical time lines and distinct events as icons. Using filtering, highlighting and interaction tools a user can visually identify similar patterns between the time lines.

The approaches described this far for identifying patterns of events are purely visual and therefore sensitive to data size and subject to human error. Therefore visualization is often combined with automatic or semi-automatic techniques that can aid visual inspection. For example an extension of Lifelines was presented in [24] where multiple medical event histories can be visually explored and similar patterns can be revealed by aligning the data on discrete events. In [8] an interface is presented for finding temporal patterns across multiple histories by allowing users to construct complex queries and search the data for matches. A disadvantage of this method is that the creation of the queries is complicated, also the user must have some knowledge about the data in order to know what to look for, hence this framework is not well suited for identification of unexpected patterns. In [23] an apriori based algorithm was implemented for mining frequent sequences of activities in activity diary data. The results are visualized in a time geographical representation for the user to explore. The disadvantage with this method is that if the search is constrained then it overlooks infrequent sequences, while if no constraints are set then it identifies large numbers of patterns, many of which are trivial and uninteresting.

Event data can be visualized using directed graph representations, where each event constitutes a node in the graph and each transition from one event to the next constitutes a directed edge. Patterns in such representations are identified by considering the structure of the graph. Examples of the identification of patterns within graphs can be found in state transition and social network analysis. The nature of state transition data is closest to the activity data considered in this paper. In state transition systems it is interesting to locate sequences of states, similar to our interest in sequences of activities. Examples of methods for visualizing and exploring state transition systems using graph representations are found in [22] and [19]. The work in [20] is the most closely related representation to our work that we have found. The state transition system is successively clustered based on node and edge attributes. The clustered nodes are then represented twice in the visualization: once as source and once as target clusters, and labelled edges are positioned in between. A user can select a source, target or edge and explore their connection patterns. The visual representation used in [20] is rather complicated and even though it is certainly effective for the expert user, it can be complicated to perform a free exploration with no predetermined goal and the display tends to get very crowded due to the complex nature of such a system. In contrast, the approach described in this paper uses a tree-like representation to explore paths of nodes one at a time, leaving the display clear from clutter, and a separate display to show how the explored paths are distributed in the context of the total dataset.

2.2 Algorithmic sequence mining

The automatic discovery of interesting sequences in large sequential data sets has been an area of substantial research and there are many different mining approaches to this general problem depending on the type of data under study. The use of visualization in combination with such approaches is not as extensive, although several examples exist. We will, in this section, briefly present the methods most relevant to our research within social science diary data, namely three different approaches to frequent pattern mining, and sequence alignment.

Frequent pattern mining has been extensively researched since it was first introduced in 1993 [2] for the discovery of patterns in large transaction databases, so called market basket analysis. In 1995 [3] the approach was extended to apply to sequential data. An excellent overview of the frequent pattern mining research status is available in [12], hence we refer the interested reader to this work for details and will just present the methodology behind the most prominent approaches for sequential data of which there are three: the apriori method [3], pattern growth [17], and a vertical data format approach [26]. All three were initially applied to frequent patterns in transaction databases and later extended to sequential and structural patterns.

The apriori approach [3] is based on the property that a sequence of k items cannot be frequent unless all of its sub-sequences are frequent. So candidate sequences of increasing length are generated, stepwise, and checked for a minimum frequency against the database. The problem with this approach is that, even though the search space is greatly reduced, there can still be a very large number of candidate patterns that have to be pruned during the database scan. Also this method requires a new database scan for each new order, k, of sequences which is extremely time consuming. An example of this approach in combination with visualization for identifying, exploring and filtering patterns in social science diary data can be found in [23].

The pattern growth approach [17] mines frequent sequences without using candidate generation. Instead each of the set of frequent single items is used as a prefix and the database is compressed into a tree having a branch for each of the prefixes. Projected databases are created for each of the single items and patterns are then mined recursively in each of these separate databases. This approach significantly reduces the number of database scans needed, at the cost of a high memory load due to the large number of projected databases needed.

The vertical data format approach for mining sequences was introduced in [26] and is a method based on the apriori property for candidate generation and testing. However, during the initial database scan the data is transformed into a vertical format, meaning that for each single item a tuple with information about the locations in which the item appears in the data is saved. The great advantage of this method is that the database need only be scanned once, since all the information needed to evaluate the frequency of any candidate sequence is then available. The drawback here is, similar to apriori, that a large number of candidate patterns have to be tested and also the method does not apply to the identification of outliers. A visualization application using this approach on web log data is described in [25].

A fourth approach that is popular for mining sequential patterns in social sciences is optimal matching [1] or sequence alignment, originally developed for protein or DNA sequences [5]. Sequences are pairwise aligned and their similarity is computed. The results can then be classified using clustering algorithms to reveal trends in the sequential event data. An example of a visualization application based on this approach can be found in [10]. This approach is effective but is conducted in a pre-processing step and can be time consuming depending on the number of compared sequences.

The main drawback of the described methods is that they primarily search for frequently occurring sequences in order to reduce the search space with the risk that potentially interesting infrequent patterns and outliers are disregarded in the process. Our goal is to allow the user to decide on the interestingness of a sequence depending on the focus of their analysis and provide the flexibility to find all types of patterns. Furthermore, these methods identify patterns in a pre-processing step and the results are then made available to the user for exploration and filtering. The danger is that a great deal of computation is performed to identify a huge number of results most of which are completely uninteresting to the user performing the analysis.

The approach that we propose puts the user in direct control of the sequence identification process, requiring no pre-processing and no post-filtering. The patterns found at each stage are visually displayed with supporting data both as the ActiviTree representation and, simultaneously, in the context of the subjects' lives. The data mining algorithm is used as a tool which provides support to the process by ranking the potential sequences at each stage of the exploration, but presents all available options, ignoring nothing, and so even infrequent patterns and outliers are available for the user's consideration.

3.1 Social Science Diary Data

This research is concerned with exploring sequences of event data in social sciences. The data we are interested in consist of activities performed during the day by individuals in a population. Time diaries kept by a group of individuals have been collected and coded into a data set of distinct activities. Each individual's day is described by a linear sequence of activities. Each activity has a unique code, a start and an end time. This data can be considered as a directed graph between nodes, the activities, via edges, the transitions between the activities. Using this metaphor the order of the performed activities is preserved even though the actual timings of each are lost. Considering the data as being such a graph allows the use of algorithms originally developed for searching the internet, such as pagerank [16] and hubs & authorities [13], which act on such graph data. A generalization of hubs & authorities is used as a starting point in this work.

3.2 Generalization of Hubs & Authorities

Kleinberg in [13] proposed a method for retrieving web pages relevant to a given query which is based on the link structure of the Web. To do this he introduces the concepts of hubs and authorities through a mutually reinforcing relationship:

"a good hub is a page that points to many good authorities and a good authority is a page that is pointed to by many good hubs"

Using this relationship within an iterative algorithm a hub and an authority score are assigned to each page and updated at each iteration of the algorithm. If a page points to many pages with large authority scores then it should get a high hub score and if a page is pointed to by many pages with large hub scores it should get a high authority score.

Given a posed search query a collection of hyperlinked pages that are relevant to this query is first identified using a standard method such as query string matching. The ordering of the pages is then determined using the mutually reinforcing relationship. The collection of hyperlinked pages related to the query can be represented by a directed graph G =(V,E) where vertex set, V, is the set of all pages, i, and edge set, E, is the set of all directed edges, (i, j), corresponding to a link from page i to page j. If h j and a j are the hub and authority scores of vertex j then the operations updating these scores are: TeX Source $$h_j \leftarrow {\rm{ }}\sum\limits_{i:(j,i) \in E} {a_i } {a_j} \leftarrow {\rm{ }}\sum\limits_{i:(i,j) \in E} {h_i } $$ As the number of iterations increases the scores converge. The resulting scores can then be ordered and filtered to return the largest scoring pages from each as the most relevant hubs and authorities.

Blondel et al. in [4] generalize the hub & authority scores to being similarity scores between the structure graph

hub → authority

and the graph, G, of hyperlinked pages. So, the authority score of vertex j in G can be seen as a similarity score between vertex j and vertex "authority" in the structure graph and the hub score of vertex j as a similarity score between vertex j and vertex "hub". Given this generalization the same mutually reinforcing iterative algorithm can be extended to apply to arbitrary structure graphs.

Consider a graph G =(V,E) and the structure graph

in → centre → out.

Three scores are now assigned to each vertex j in G, an 'in-score' (xⁱⁿ_j), a 'central-score' (x^c_j) and an 'out-score' (x^out_j), and the updating operations of these scores are: TeX Source $$\eqalign{x_j^{{\rm{in}}} \leftarrow {\rm{ }}\sum\limits_{i:(j,i) \in E} {x_i^c } \cr x_j^{\rm{c}} \leftarrow {\rm{ }}\sum\limits_{i:(j,i) \in E} {x_i^{in} } {\rm{ + }}\sum\limits_{i:(j,i) \in E} {x_i^{{\rm{out}}} } \cr x_j^{{\rm{out}}} \leftarrow {\rm{ }}\sum\limits_{i:(i,j) \in E} {x_i^c } \cr} $$ The adjacency matrix B of graph G can be retrieved by setting each entry (i, j) of B equal to the number of links from vertex i to j. The updating operations can then be written in matrix form: TeX Source $$\left[\matrix{x^{{\rm{in}}} \hfill \cr x^{\rm{c}} \hfill \cr x^{{\rm{out}}} \hfill \cr} \right]_{k + 1} = \left[\matrix{0{\rm{ }}B{\rm{ 0}} \hfill \cr B^T {\rm{ 0 }}B \hfill \cr {\rm{0 }}B^T {\rm{ 0}} \hfill \cr} \right]\left[\matrix{x^{{\rm{in}}} \hfill \cr x^{\rm{c}} \hfill \cr x^{{\rm{out}}} \hfill \cr} \right]_k,{\rm{ }}k = 0,1,...$$ This can be written in compact form: TeX Source $$\eqalign{ x_{k + 1} = {\rm{ }}Mx_k,k = 0,1,... \cr {\rm{where}} \cr x_k = \left[\matrix{x^{{\rm{in}}} \hfill \cr x^{\rm{c}} \hfill \cr x^{{\rm{out}}} \hfill \cr} \right]_k,M = \left[\matrix{ 0{\rm{ }}B{\rm{ 0}} \hfill \cr B^T {\rm{ 0 }}B \hfill \cr {\rm{0 }}B^T {\rm{ 0}} \hfill \cr} \right] \cr} $$ and k is the iteration. Since it is the relative strength of the scores that is interesting, the normalized score values, z, can be used: TeX Source $$z_0 = x_0 > 0,z_{k + 1} = {{Mz_k } \over {||Mz_k ||}},k = 0,1,...$$ Any positive vector x ₀ could be used to initialize this process. In this work we use the unit vector. As the number of iterations increases, k → α, the score vectors, z₁,z₂,z₃,…, ideally converge. However, the authors of [4] point out that z rarely converges to one limit but rather oscillates between an even and an odd limit. The even limit, given by: TeX Source $$z_{{\rm{even}}} = \mathop {\lim }\limits_{k \to \infty } z_{2k} $$ is selected as the definition of the similarity scores. The highest scoring vertices then constitute the most significant in, central and out nodes in the graph.

Considering our activity data as a directed graph, A =(V,E), with vertex set, V, being the activity set and edge set, E, the set of activity transitions, this described methodology can be applied in order to assign significance measures to activities and successively explore activity patterns in a population. This is described in the next section.

3.3 Interactive Sequence Identification

In this paper we incorporate the generalization of hubs & authorities [4], described in section 3.2, into a visual data mining interface for exploring activity data and identifying patterns of activities. A pattern in the activity diary data context is a sequence of activities (1 ,2 ,…,n) that, combined, express some interesting behaviour.

To give an overview, our method works as follows. The data is scanned and the significant single activities (1-sequences) are identified, these constitute the patterns of order 1. Selecting a 1-sequence as a starting point, significance scores for each activity connecting to this sequence are calculated. Connecting activities are then drawn in a tree-like representation (figure 1(a)) in which the currently explored starting pattern is drawn in the middle and activities connecting to it are drawn as nodes linking into or out of it. Visual cues are given for significance and frequency of each node. A user can steer the exploration by choosing a connecting node to add to the identified 1-sequence creating a 2-sequence. This starts a new iteration of the algorithm and the nodes connecting to the new sequence are drawn. Having the user steer the creation of the sequences means that no pre-processing is needed. Also the algorithm does not need to have previously identified all n−1 sequences in order to find the n sequences. Given any sequence it will immediately calculate the significance of the activities connecting to it. We will now describe the details of the process.

3.3.1 Algorithm

The algorithm for identifying significant n-sequences has three steps:

1. Creation of an adjacency matrix.

2. Repeated application of formula 5 to get similarity scores.

3. Weighting of the retrieved scores.

The initial step is to find all significant 1-sequences. This prepares the ground for the exploration to start.

Similarity scores are calculated between the nodes of the total activity graph, A, and the structure graph

in → centre → out

by first computing the adjacency matrix B, of A. This is done by setting each element (i, j) of B equal to the number of transitions from activity i to activity j. The rows of B then represent in-links to, and columns out-links from, each activity. The mutual reinforcing iteration, formula 5, is then used to assign 'in-', 'central-' and 'out-scores' to each activity node. We use a convergence criterion for deciding the appropriate number of iterations, the magnitude of k, in formula 5. For the data that we use this equation always converges within 10 iterations.

The similarity scores give an indication of the behaviour of each node within the graph, hence activities with a high in-score point to many significant activities, activities with a high out-score are pointed to by many significant activities and activities with a high central-score are both pointed to by, and point to significant activities. In the initial stage, a reasonable assumption to make is that the activity nodes with a high central-score are the most appropriate as patterns of single activities, 1-sequences, since these are the most prominent 'link' activities holding the overall sequence of the individuals, in the data, together. Hence we choose the central-score as the significance score for the single activities.

The computed similarity scores are a measure of significance that is dependent on the connectivity of the nodes in the graph. Weighting this measure with the frequency of occurrence of the nodes gives a more general sense of significance in the context of the activity dataset. A node, for example, pointed to many times by a single low scoring node can be equally as interesting as a node pointed to once by each of several significant nodes. However, using only the similarity scores the former would not be considered a significant node. Hence, weighting the scores by frequency of occurrence balances these factors. Furthermore, since we are interested in the relative values of the scores we normalize them to the range [0, 1]. So now activity patterns of order 1, 1-sequences, are found along with their significance scores and the exploration process of higher order patterns can begin.

In order to explore n-sequences, an (n-1)-sequence is used as a query sequence. A subgraph of the total activity graph, A, is found and the adjacency matrix, B, of this subgraph is computed. So to explore 2-sequences, for example, a 1-sequence is used as a query sequence.

The subgraph considered consists only of the currently explored query sequence itself, which can be seen as a supernode, q, and of activities linking into and out of the query sequence (figure 2). This is done by scanning the activity data set for matches of the query and considering the preceding and succeeding activities to these matches. An adjacency matrix, B, is computed by setting each element (i,q₁) of matrix B, where q₁ is the first activity of the query sequence, q, equal to the number of transitions from activity i to the query sequence and each element (q_n−1, j) of B, where q_n−1 is the last activity of the query sequence, q, equal to the number of transitions from the query sequence to activity j. The rows of B then represent in-links to, and the columns out-links from, the query supernode. Figure 3(b) shows an example of such a subgraph adjacency matrix.

Having computed the adjacency of the subgraph the mutual reinforcing iteration, through formula 5, is applied in the same manner as before and 'in-', 'central-' and 'out-scores' are assigned to each activity node connecting to and from the query sequence. The objective now is, given an (n-1)-sequence query, to explore all possible n-sequences. So, we are interested in finding the activities linking into and out of the query and their respective scores, this means that we consider the activity nodes twice. Once as in-nodes, which are the activities having high in-scores linking into the query and once as out-nodes which are those with high out-scores linking out of the query.

The weighting of the scores also occurs in two steps. All in-nodes are weighted by the number of times each of them appears as an in-link into the subgraph, their frequency of 'in occurrence', while the out-nodes are weighted by the frequency of their 'out occurrence'. The weighted scores are then normalized to the range [0, [1]. As a result we get all nodes linking into the explored query ((n-1)-sequence) and their significance score (in-score) and all nodes linking out of the query and their significance score (out-score) and are now able to explore the combinations that can be made by adding these nodes to the query.

Fig. 2. Example of links into and out of a query supernode q.

Previous | View All | Next

Fig. 3. Example representations of adjacency matrices. (a) Adjacency matrix of the total graph of activities. Each element (i, j) in the matrix represents a transition from i to j. (b) Adjacency matrix of a subgraph of activities showing how these connect to a query sequence q. Each row element, (i,q₁), represents a transition from i to the first element in the query sequence, q₁, and each column element, (q_n−1, j), represents a transition from the last element in the query sequence, q_n−1 to j.

Previous | View All | Next

Alternative weighting by features of the data can be combined into the procedure in two ways, depending on the nature of the weighting to be applied. The adjacency values can be modified for each activity transition to reflect it's desirability. This is then combined into the calculation of the scores for each potential node. Alternative weightings can also be combined into the scores themselves at each iteration.

To summarize, the algorithm consists of three simple computational steps. An adjacency computation, a repeated matrix-vector multiplication and the application of weighting factors. The process of computing the adjacency matrix, B, expands linearly with the number of transitions between activities and so with the number of activities recorded in their diaries by the participants in the study. The dimension of the matrix, M, and the scores vector, x_k, in equation 5 are defined by the number of distinct nodes in the activity graph. In the case of our social science activity diary data this is a fixed value of approximately 330 different activity codes. Hence this step takes constant time and M occupies approximately 8MB. The matrix is very sparse so, in cases where the size of the matrix is much larger, sparse matrix methods could be used. The simplicity of these operations is what makes the exploration process highly interactive. Furthermore, any rejection of identified sequences is performed by the user. There is no cut-off by the algorithm, the identified sequences are simply ordered by their significance and made available to the user for filtering and analysis.

3.3.2 ActiviTree

The sequence exploration algorithm is steered using an interactive visual interface in which the identified activity sequences performed by individuals in their daily life are represented in a tree-like fashion which we therefore call 'ActiviTree' (figure 1(a)). The ActiviTree is implemented as a feature within a visual exploration tool for activity diary data, details of which can be found in [7], [23]. In order to start exploring patterns in the data the significance of the single activities is initially computed and the results are presented to the user in an ordered list format, see figure 4. Clicking on an activity in the list will pop up the ActiviTree and the exploration process can start. In the ActiviTree time goes upwards.

Fig. 4. Part of the list of identified single activities.

Previous | View All | Next

The currently explored query activity sequence is drawn in the middle (as the trunk of a tree), the activities preceding it are drawn as nodes pointing to the query sequence (as roots connecting to the trunk) and the activities succeeding it are drawn as nodes pointed to by the query sequence (as branches growing from the trunk). Both in and out activity nodes are ordered by their significance score from left to right. So the leftmost activity is the most significant one. The frequency of their occurrence is mapped onto the opacity of the edge connecting to the query, so frequent sequences are opaque while infrequent ones appear more transparent. Each activity node has a label that includes its corresponding activity code and its frequency of occurrence as an in/out-node to/from the query. Right clicking on an activity node will show an explanation of the activity code. Nodes are colour coded in accordance with the activity classification colour scheme in [7].

Clicking on an in or out activity node in the interface will add it to the query sequence and identify the in- and out-nodes of the new query sequence in a fraction of a second. Clicking on the head or tail of the query sequence will remove the selected node from the query and update its in- and out-nodes.

If too many nodes connect to the explored query the representation can become cluttered and the nodes overlap. We avoid this by implementing a decluttering mode which shrinks the edge length of the nodes and expands a few at a time on mouse over (figure 5). Clicking on the nodes is available in this mode also, so the user is still able to select a node to add to the query and continue the exploration.

Each time a node is added to the currently explored query sequence the identified new activity sequences and their locations in the dataset are saved. Removing a node from the query sequence, and choosing a different exploration path will replace the saved sequences and their locations. The identified activity sequences can be highlighted in a separate linked view within the context of the individuals' days in order to study their distribution more closely. This can be seen in figure 1(b) where the query sequence of figure 1(a) is highlighted. In the separate linked view time is represented on the y-axis going upwards and the individuals are represented ordered on the x-axis by sex and age. So, from left to right one can see older men to young boys and older women to young girls. This view reveals the distribution of the explored sequence across ages and sexes but also the duration of the sequence, its repetitiveness, and its time of occurrence during the day.