Dynamic Bayesian Network Modeling, Learning, and Inference: A Survey

Since the introduction of Dynamic Bayesian Networks (DBNs), their efficiency and effectiveness have increased through the development of three significant aspects: (i) modeling, (ii) learning and (iii) inference. However, no reviews of the literature have been found that chronicle their importance and development over time. The aim of this study is to provide a systematic review of the literature that details the evolution and advancement of DBNs, focusing in the period 1997–2019 that emphasize the aspects of modeling, learning and inference. While the literature presents temporal event networks, knowledge encapsulation, relational and time varying representations as the four predominant DBN modeling approaches, this work groups them as essential techniques within DBNs and help practitioners by associating each to various challenge that arise in pattern discovery and prediction in dynamic processes. Regarding learning, the predominant methods mainly focus on scoring with greedy search. Finally, our study suggests that the main methods used in DBN inference extend or adapt those used in static BNs, and are oriented to either optimize processing time or error rate.


I. INTRODUCTION
Probabilistic Graphical Models (PGMs) use a graphical representation to compactly express probability distributions while at the same time explicitly represent large joint distributions, for transparent evaluation by specialists [1]. According to [2], PGMs can be classified into: (i) directed/undirected, (ii) static/dynamic, and (iii) probabilistic/decisional. The first group represents symmetric (undirected) or asymmetric (directed) dependency relationships. The second group represents a set of variables at a specific point in time (static) or across a period of time (dynamic). The third group uses random variables (probabilistic) or decision and utility variables (decisional). Among the different dynamic representative PGMs we have (1) Markov Chains, (2) Hidden Markov Models, (3) Markov Decision Processes (MDPs), (4) Partially Observable MDPs and (5) Dynamic Bayesian Networks (DBNs). Markov Chains [2] are (i) directed and (ii) probabilistic models that present discrete numbers The associate editor coordinating the review of this manuscript and approving it for publication was Hao Ji. of states and transitions that are stochastic. Hidden Markov Models are also (i) directed and (ii) probabilistic and are comprised of a double stochastic process where one set is underlying and unobservable and only revealed through a sequence of observations from the second set of processes [3]. Markov Decision Processes (MDPs) are (i) directed and (ii) decisional sequential decision models that evolve over time and are controlled by an agent [4]. Partially Markov Decision Processes are (i) directed and (ii) decisional but differ from MDPs as they are designed to address hidden or partial information concerning the state of the system [2]. Finally, Dynamic Bayesian Networks (DBNs) are extensions of Bayesian networks to model dynamic processes and consist of a series of time intervals that present the states of all variables at a given time and thus represent the evolution of a process over time [1]. As such, DBNs can be seen as a generalization of Markov Chains and Hidden Markov Models because they represent a space of states in a factorized way instead of as a single discrete random variable [5], and can be classified as (i) directed and (ii) probabilistic. Also, DBNs can represent a linear dynamical system such as Kalman filters, where the VOLUME 9, 2021 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ variables are all continuous and all of the dependencies are linear Gaussian [1]. DBNs are important as they capture and analyze information over time and fulfill two important functions in machine learning: classification and pattern discovery. Other classification algorithms, such as neural networks act as black boxes and make it difficult for a specialist to interpret the resulting model according to the domain of the problem, DBNs on the other hand are advantageous as they transparently encoded probability distributions over complex domains [1]. Some examples of applications that require the use of DBNs in order to capture their dynamic behavior are the pattern detection of human brain behavior [6], speech recognition [7], [8], medical diagnosis [7], [9], identification of regulatory gene networks [8], [10], target tracking [8], visual activities [11], [12], crime risk analysis [9], [12], sensor validation [9], client analysis [9], and video object tracking [12].
DBNs have three important aspects [2]: (1) modeling, (2) learning, and (3) inference. Regards to modeling, concepts from other domains have been merged with DBNs in order to present a better representation of the data and its behavior. To achieve this, studies have proposed Temporal Event Networks (TEN) [13] or relational ones [14]. Modeling, in general, has gone in the direction of specialization instead of generalization, resulting in useful models for specific contexts. Automatic learning has emerged as a response to the automation of the representation of DBNs, inspired particularly by static BNs. Search and parameter estimation algorithms typically found in BNs are also used in DBNs. As stated previously, the main challenge in highly dense network DBNs is to construct them efficiently [15]. One of the prevailing professional applications of DBN construction has been in the biomedical area [16], [17]. Inference allows the DBN to develop a diagnosis or prediction while maintaining efficient execution time and memory usage [18]. Automatic learning allows the DBN to build networks capable of representing relationships from data. Modeling allows the DBN to analyze dynamic behavior on a granular time scale. According to [1], there are two problems that arise when using DBN inference, as opposed to BN inference. One is that the BNs generated from the DBNs can have an arbitrarily big and complex structure. The second problem is that the temporal reasoning is often different from the reasoning required of a static model. This is especially common in networks with dense connectivity that pose problems for exact and approximate inference algorithms and thus require an algorithmic analysis. Variants of exact inference algorithms based on the variable elimination were proposed long before the development of probabilistic graphical models [1].
Despite the importance of DBNs and the diversity of studies and aspects developed, a Scopus or Web of Science (WoS) level literature review could not be found in the literature. While a Google Scholar search identifies two technical reports between 2001 and 2006 [19], [20], new solutions have been described to address the diversity of challenges associated with inference, automatic learning and modeling.
To address this gap, this paper aims to answer the following question: What advances have been made with respect to modeling, automatic learning and inference in DBNs?
In the reminder of this paper, we carry out a literature review on DBNs with the following structure. In Section II, we present the background and theoretical basis of DBNs. In Section III, we present the methodology of the literature review. In Section IV, we analyze the different DBNs and present their approaches in detail. In Section V, we discuss the findings and important characteristics of the DBN studies. Finally, in Section VI, we detail the conclusions of the work.

II. BACKGROUND
Definition of DBNs are described in detail in [1], [5]. X t = {X t 1 , . . . , X t I }, denotes a set of random variables representing the state process at a given time t. A DBN is a pair (G, θ), where G is the structure and θ is the set of parameters of a DBN. The DBN models a dynamic process, specifying a probability distribution for X 0 , . . . , X T with P(X 0 , . . . , X T | G, θ). G is a directed acyclic graph (DAG), whose nodes are the variables X 0 , . . . , X T , whose edges follow a dynamic sequence defined as X t → X t+1 , where t ∈ {0, . . . , T − 1}, and cannot have edges of a future time pointing to a past time of type X t ← X t+1 . θ is a set of parameters that contains a conditional probability distribution P( Modeling is based on (G, θ) representation, a basic property of any model, having (1) the entities that constitute it, and (2) the relationships between these entities. All probabilistic network models are represented as graphs to define their structure and with local functions to describe their parameters. The difference between one model and another is the type of graph and the local functions used. The G structure is designated as a model and the θ distribution is designated as the parameters. In most cases, it is assumed that a DBN presents the same model at every time t. In that sense, this model is dynamic as its parameters vary over time and their distribution is estimated each time a new observation occurs.
Learning consists of building models in one of two manners: (1) by hand with the support of specialists, and (2) automatically from data. The current trend is to use automatic learning techniques. In many cases, estimates of G, θ use automatic learning techniques, also known as Bayesian learning. This technique consists of calculating the probability of each hypothesis from the data. If D is a data set and h i is the ith hypothesis, it is possible to estimate the probability of each hypothesis that maximizes P(h i | D), using the maximum a posteriori (MAP) estimate.
Inference consists of answering the probabilistic query according to the model and a set of evidence. The inference is a basic task to compute the posterior probability distribution for a sert of query nodes, given values for some evidence nodes which is called belief updating or probabilistic inference [21]. The representation G, θ responds to queries through the intractable process of inference, and can be defined as the calculation of the probability distribution a posteriori of a set of query variables given a set of observed events. Let a query variable Q and a set of n evidence variables denoted by E = {E 1 , . . . , E n }, represent observable events E = e, where e is the evidence. To respond to the query, the conditional distribution P(Q | e) is used.
DBNs intend to reveal patterns and temporal relationships by capturing the complexity and variable nature of a problem. DBNs model these dynamic processes, naturally establishing a compact structure capable of capturing the semantics of the temporal relationships between measured events within a dynamic system.

III. RESEARCH METHODOLOGY
Similar to [22], in order to query the literature for reviews on DBNs, we used the following keyword search strings in both the Scopus and WoS citation databases: No results were found. We performed a second search in Google Scholar with the query: (''dynamic bayesian network'' AND (''survey'' OR ''review'')) . which retrieved two articles from the years 2001 and 2006.
In this literature review, the standard systematic review methods for software engineering area were considered [23], [24]. The final method was divided into three phases: (1) planning, (2) development, and (3) results. In the planning phase, the importance of reviewing the literature is discussed, the research questions are formulated, and the study protocol is presented. In the development phase, the primary studies are evaluated for potential inclusion and filtered for data extraction. In the results phase we present statistics and findings and answer the research questions posed in the first phase. The planning and development phases are described in this current section III, while the results phase consisting of statistics and answers to the research questions, are found in the following sections.

A. PLANING OF THE REVIEW
This study is designed to answer three specific research questions concerning the progress of DBNs (Table 1). The search process is carried out by designing a search string that queries a citation database (Table 2). For the design of this search string, it is important to use synonyms associated with DBN terms that best represent the research questions. This review covers the relevant research from 1997 to 2019, using the Scopus and Web of Science citation databases. The selection criteria are shown in Table 3.

B. DEVELOPMENT OF THE REVIEW
Once the research questions and selection criteria were defined, the article search process was implemented.

IV. ANALYSIS
In this section, answers are given to the research questions posed in Section III-A. TEN simplifies the DBNs into small dynamic processes. In traditional DBNs, a node represents the value of a variable at a certain time, while in TEN, a node represents when an event or change of state of a variable occurs, and thus it is simpler and more efficient than DBNs [2].
TNBN, TBNDTE and DTPEN are three subtypes of NET [13], [25] and are more efficient than DBNs for small problems. However, DTPEN has a disadvantage of having a single time granularity [26]. The studies involving these subtypes of TENs are shown in Table 5.
Two types of DBN representations were identified as a way to encapsulate knowledge ( Table 6): Dynamic Influence Network and DBN (DIN-DBN), and Ordinary Differential Equations in DBNs (ODE-DBN). While Dynamic Influence Network (DIN) presents a compact modeling procedure that permits efficient managing of temporal restrictions [41], it does not assimilate updated information easily, a challenge which has produced new methods that would alternate between it and DBN inference in a single process [28]. This has also been achieved with differential equations (ODE-DBN) [29].    The modeling of relational DBNs has non-directed edges unlike traditional DBNs, which are applied to scenarios where relationships are bi-directional, such as friendship relationships in a social network. An inventory of relational DBN modeling is shown in Table 7.
One of the issues addressed in recent years concerns the intractable challenge of implementing algorithmic methods that allow DBN structures and parameters to evolve over time, since it involves making complicated update, change interval, and network structure design decisions, among others (Table 8). Figure 4 illustrates the emergence of the DBN modeling types over time.

B. RQ2: WHAT ADVANCES HAVE BEEN MADE WITH RESPECT TO DBN LEARNING?
The literature presents many different approaches that describe effective learning methods (Table 9), all of them can be organized into four groups of strategies for efficient  building of DBNs (Table 10). The first strategy is scoring or greedy search, functions that use metrics to measure the quality of each structure in the structure search space. The second is constraints, a strategy that applies statistical techniques to restrict the use of edges within the graphical structure. Sampling is the third group, which allows for the generation of possible structures from a distribution. The final strategy is a posteriori probability, which generates structures after having validated their usefulness (see Table 11).
Concerning the construction of their structures, the reviewed studies present two important factors that affect learning and ultimately the DBN's proper performance. One is the quantity of random variables a factor that reduces performance as its quantity increases. Another key factor in DBN learning performance is the relationship between variables that must be filtered by causality methods. Therefore, managing hundreds of variables when building the structures of the networks requires very efficient algorithms. However, a large quantity of variables also requires a valid connection through edges by means of causality measures. If machine learning is used, this becomes even more complex, considering the search space of directed acyclic graphs.
Studies suggested that these learning methods are best evaluated using metrics that consider their structure, the structure generation time, the inference results using the learned structure, and the time intervals (Table 10). The evaluations considering structure look to determine the one that best represents the problem, derived from the data, and are mainly used to build regulatory networks in the biomedical field. The evaluations considering the structure generation time intend to measure how long it takes the method to build a DBN structure. The evaluations that consider the inference using the produced structure are based on its performance of tasks, such as classification and others. Finally, the evaluations regarding time intervals intend to measure the ability of the method to select the adequate time intervals and their corresponding time-slices for the generation of a DBN.
Regarding the types of datasets utilized, 15 works use synthetic datasets while 13 utilized real datasets, with biomedical predominating the studies with real datasets (Table 12).

C. RQ3: WHAT ADVANCES HAVE BEEN MADE WITH RESPECT TO DBN INFERENCE?
Six studies were found relating to the inference aspect of DBNs and collectively identify three types of inferences ( Table 13). One type, exact inference, intends to estimate a query in terms of the conditional probability P(Y | X), where Y is the random variable to be estimated and X is the evidence and is considered a NP-hard problem that requires development in exponential time in the worst case [1]. In [54], a structural interface algorithm is presented that accelerates  inference by exploiting the repeated and local structures as well as the conditional independences, thus improving their scalability for large and complex networks.
Approximate inference intends to find a successful solution in the shortest time possible. Its approach uses the DBN factors to estimate the joint distribution and, in many cases, use additional information to support the inference [55]. Carrying out inference on the different time-slices is a challenge, and requires important design considerations, such as whether to include supporting information and which DBN segment should be applied to perform the inference. Hybrid inferences are those that combine the characteristics of both exact and approximate inference, enabling them to develop selective updates about the belief factors from the network, and thus producing exact inference under certain assumptions, and approximate inference under others [56].
For optimum performance, inference methods look for efficiency in at least one of the following metrics: processing  time, error rate (minimum certainty limit), coherence, and scalability. Coherence refers to the consistency of results from the logical use of data in the specific domain addressed, while scalability refers to the suitability of these methods for large networks, with processing time and error rate being the two metrics most commonly used (Table 13).

V. DISCUSSION
The result of this systematic review is a catalog of factors that influence the building of DBNs. Researchers can use the different metrics, strategies and criteria presented herein to understand and determine the optimal approaches for their specific application. The relevance of this information is validated as 90% of the reviewed studies were from the first and second quartiles (Q1, Q2) journals and thus fortify the findings presented in this review. Each research question is discussed below.

A. RQ1
it was identified that, in general, inference in DBNs involves to estimate probabilities to answer queries from the representation, transition and observation of the network. Unlike static BNs, DBNs include the transition step, which allows the sequential transfer of probabilities between different time periods. The reviewed studies show a greater tendency to apply approximate inference as opposed to hybrid inference, possibly due to its low computational cost. Inference evaluation metrics are diverse, and include consistency, scalability, VOLUME 9, 2021 time, and error rate, with the last two being the most commonly used. However, there is no consensus on which is the most informative metric to use and few studies apply them all. Inference methods for DBNs are mainly adaptations of static BNs.

B. RQ2
Regarding the learning process, studies were found on the construction of the structure of the Bayesian network G =< V , E >. They also showed that a majority of the technical studies on learning strategies used scoring and greedy search methods, while a minority used constraints, a posteriori probability, and sampling. In addition, these strategies are oriented to (1) machine learning and (2) experiential learning. Machine learning deals with problems such as scarcity of data, unhelpful relationships between variables that decrease performance, or very wide search spaces where it is not possible to achieve the convergence of an optimal model. These limitations suggest that further research on new learning methods that improve the performance of DBNs for both classification and pattern discovery is warranted. However, there are strategies that have not been applied to DBNs, nor even BNs. One is using GRASP instead of greedy search. Also, the evaluation approach was oriented towards optimizing the quality of the structure of the results, the construction time, and the time intervals for the structure. For a fixed number of instances, the quality of results declines as the quantity of instances increases, while the quality of results improves when the relationships are filtered by some measure of causality in E. However, no studies were identified that address both aspects at once.

C. RQ3
New approaches to DBN modeling have emerged, such as temporal event networks, knowledge encapsulation, relational, and time-varying. In time event networks, the objective is to simplify the construction of a DBN in order to evaluate events in small dynamic processes. In knowledge encapsulation, differential equations or dynamic influence networks are used to represent data. In relational modeling, the objective is to manage cyclicality in order to include it within the modeling of the DBMs, taking advantage of the changing processes in time without affecting the temporal reasoning. In time-varying models, the objective is to consider the evolution of the structure and parameters of the DBNs over time, with a clear interest in seeking DBNs that contribute to the monitoring of complex domains as time progresses. This is an important topic and is likely to continue receiving greater attention in the coming years due to its impact on monitoring applications.

VI. CONCLUSION
This work aimed to provide a systematic review of the literature related to new approaches of inference, learning and modeling of DBNs. Three research questions were proposed regarding the advances in inference (RQ1), learning (RQ2) and modeling (RQ3). The search was carried out in both the Scopus and Web of Science citation databases, selecting 42 out of 777 identified studies, with 42.8% of those studies addressing learning (RQ2) and 47.6% addressing modeling (RQ3). It is important to point out that 90% of the selected articles belong to journals from the first and second quartiles (Q1, Q2), which ensures that this study presents reliable results. With regard to modeling, it is important to emphasize that DBN modeling approaches are evolving, with four predominating: (1) temporal event networks, (2) knowledge encapsulation, (3) relational, and (4) time-varying. However, as no studies were identified that involve more than one approach at a time, future research should consider this as it could bring good results at low computational cost. Specifically regarding modeling, this work seeks to contribute to the practice by grouping these four modeling aspects as essential components of DBNs and associating each to the various challenges found in pattern discovery or prediction in dynamic processes to further advance the effectiveness of DBNs. With regard to learning, studies related to structure learning and its evaluation were identified. Learning methods are oriented to scoring with greedy search and the reviewed studies show that the quality of results declines as the number of variables increases while the quality of results improves as the number of edges increase. Future research should seek to understand the effect of these two aspects at the same time, as well as apply more advanced search strategies, such as GRASP. About inference, it involves the probability estimation to answer queries from the representation, transition and observation in the network. The identified methods are an extension or adaptation those used in static BNs and usually tend to consider the approximate approach over the exact and hybrid ones, possibly due to their lower computational cost. While time and error rate are the most commonly used metrics to evaluate the methods, no studies were identified that focused on optimizing both metrics at the same time. A limitation of the study is that it analyzed only Scopus or Web of Science studies, leaving aside conference articles and other sources that could shed more light on the benefits of Dynamic Bayesian Networks.