A Dual Learning Model for Vehicle Trajectory Prediction

Automated vehicles and advanced driver-assistance systems require an accurate prediction of future traffic scene states. The tendency in recent years has been to use deep learning approaches for accurate trajectory prediction but these approaches suffer from computational complexity, dependency on a specific environment/dataset, and lack of insight into vehicle interactions. In this paper, we aim to address these limitations by proposing a Dual Learning Model (DLM) using lane occupancy and risk maps for vehicle trajectory prediction. To understand the spatial interactions of road users, make the model independent of the environment, and consider inter-vehicle distances, we embed an Occupancy Map (OM) into the trajectory prediction model. We also utilise a traffic scene Risk Map (RM) to explicitly consider a comprehensive definition of risk based on Time-to-Collision in the traffic scene. These two features employed in the encoder-decoder architecture improve system accuracy with less complexity and provide insight into the interaction between all road users. The experiment has been conducted on two different naturalistic highway driving datasets (i.e., NGSIM and HighD) demonstrating algorithm independence from a single environment. Comparison results indicate that the DLM achieves a more accurate trajectory prediction with a less complex structure compared with existing approaches in terms of RMS prediction error, which indicates the effectiveness of DLM in such a context.


I. INTRODUCTION
Each year, 1.35 million people die, and as many as 50 million are injured and experience long-term disability in road crashes.Road crashes are the 8th leading cause of death and the biggest killer of those aged 5-29 [1].To reduce road crash fatalities and have a safer and more efficient transportation system, automated vehicles and driver-assistance systems have become a promising solution.Recent research has revealed the widespread significance of these technologies [2]- [5].Despite substantial attention of researchers and industry, accurate prediction of vehicle trajectory in a dynamic traffic scene remains a priority, challenging problem due to its complex nature.In a reliable trajectory prediction system, accuracy is not the only indicator of system effectiveness.The horizon of prediction, ability to be generalised to The associate editor coordinating the review of this manuscript and approving it for publication was Kan Zheng .new traffic environments and consideration of future vehicle trajectory as influenced by the surrounding vehicle trajectories (inter-vehicle interaction) are also significant.
In this study, we propose a Dual Learning Model (DLM) for vehicle trajectory prediction that identifies salient information in an Occupancy Map (OM) and a Risk Map (RM) to learn about inter-vehicle interaction and associated risk within the traffic scene in an unsupervised manner.It is a data driven approach with two input channels, using a Long-Short-Term Memory (LSTM) encoder-decoder structure.This methodology is motivated by the success of LSTM networks in modelling non-linear temporal dependencies in sequence-to-sequence prediction.We have evaluated the proposed system with two distinct naturalistic highway driving scenarios which vary in their traffic flow and driving style to illustrate the ability of the proposed system to be extended for different environments.The contribution of this study can be summarised as follows: i.We inject the Risk Map into the trajectory prediction pipeline, where the proposed structure learns to automatically extract, store and retrieve salient information without any supervision.ii.We test the proposed model on two distinct naturalistic highway driving datasets, containing more than 108570 unique trajectories from different datasets, to prove the independence of the proposed system from a specific environment.iii.We extensively evaluate the performance of the proposed method with state-of-the-art methods.We show that our proposed method outperforms the state-of-theart methods.The rest of this paper is organised as follows: First, we provide a brief background to the problem including a literature review followed by a discussion of the problem formulation.Next, we introduce the structure of our proposed models in Section III.Then, we present the simulation result and discussion.Finally, a summary of our numerical results is followed by the concluding remarks.

II. BACKGROUND AND LITERATURE REVIEW
Vehicle trajectory prediction is required in various areas of transportation such as automated vehicles and driving assistance systems.An extensive survey of vehicle trajectory prediction models classified trajectory prediction approaches into three categories; Physics-based models, Manoeuvre-based models, and Interaction-aware models [6].
Physics-based approaches have a short time prediction (i.e., less than a second).These approaches assume a constant speed and orientation for the vehicles and possess the lowest degree of abstraction in trajectory prediction.These applications are the most common and simplest techniques using dynamic and kinematic models for trajectory prediction [6].One of the weaknesses of these systems is that they cannot predict change in vehicle motion caused by manoeuvres or change because of external factors, for instance change to front vehicle speed.
The second category is Manoeuvre-based approaches.These approaches assume that a vehicle performs a series of independent manoeuvres from the other vehicles on the road network.The rationale for this approach is twofold; first the vehicle manoeuvre is recognised and then the result of manoeuvre recognition modules is used to make better predictions of future trajectories [3], [7].The manoeuvre recognition step is typically provided by classifiers such as Heuristic-based classifiers [8], Bayesian networks [7], hidden Markov models [3], [9], random forest classifiers [10] or recurrent neural networks [11], [12].The output of the previous manoeuvre classification step is an input to the trajectory prediction step and the output of this module is the future location of the vehicle [3], [7]- [9], [13].Although Manoeuvre-based approaches can decompose vehicle movements into several succinct manoeuvres that are relatively easy to recognise [6], in a complex scenario, it is hard to reasonably categorise vehicle manoeuvres into several categories.Furthermore, trajectories must be manually labelled for training purposes, and more so, labelling tasks is time-consuming and labelling errors may increase the error of the trained model.
The third category is Interaction-aware models.These approaches consider the efficacy of inter-vehicle interaction on vehicle trajectory prediction.They are the most comprehensive approach providing more reliable long-term prediction.Coupled HMMs (CHMMs) can model pairwise dependency of motion in complex traffic situations [14].However, when the number of possible pairwise dependencies increases quadratically with the number of entities, this complexity is not manageable.To solve this problem, Oliver and Pentland [15] proposed a solution to simplify the model by considering the individual impact of each vehicle surrounding the ego-vehicle,1 thereupon CHMMs would be asymmetric and computational complexity reduced.
Interaction-aware models based on the method of considering inter-vehicle interaction can be broadly categorised into two domains: handcrafted cost functions and end-to-end models [4].
In the first, handcrafted cost functions are used based on the relative configuration of vehicles and predict future manoeuvres by considering these cost functions [3], [16].Bahram, et al. [16] proposed a two-level interaction-aware manoeuvre prediction model.In the first level, the future manoeuvres of the surrounding vehicle are predicted by considering the road geometry, traffic rules and interaction between the vehicles in a traffic scene.Then, in the second level, a Manoeuvre-based classifier is used to learn different manoeuvre patterns, which in an unusual driving case, makes the prediction more robust against possibly inaccurate model assumptions.Although cost function-based methods can be generalised to new traffic conditions and do not rely on training data, they are bound to the design of the hand-crafted cost functions.
The second group is end-to-end models which learn from real traffic data.These models skip the manoeuvre recognition module and perform a direct trajectory prediction which can avoid errors in manoeuvre recognition [2].Owing to the massive variation in traffic configurations, these approaches require a large dataset for generalisation [4].With advances in artificial intelligence, various deep neural network (DNN) techniques have now been carried out on trajectory prediction.Among different methods of DNN, Recurrent Neural Networks (RNNs) and their variants such as the long-shortterm memory (LSTM) model have proven to work effectively with time series data sequence generation.One of the most popular methods is using an encoder-decoder structure to encode and decode trajectory information.Park, et al. [17] proposed an encoder-decoder LSTM model for predicting vehicle trajectory by using an occupancy grid map [18] and a beam search algorithm which reduces the error propagation issue appearing in the iterative decoding step and keeps the best trajectories for each decoding iteration.
The maximum prediction horizon of this model is two seconds, however literature shows that this timeframe might not be sufficient for applications such as collision warning systems where the optimal timing of prediction is at least 4 seconds [19].
Deo and Trivedi [4] adopt a convolutional social pooling LSTM based model.This model contains an LSTM encoder which takes the most recent track history of the ego-vehicle and all the surrounding vehicles to learn vehicle motion dynamics.A convolutional social pooling layer extracts a social tensor which captures interdependencies of vehicles.Finally, a manoeuvre-based decoder predicts a distribution of future vehicle trajectories.This approach first predicts the manoeuvre of a vehicle and then predicts the trajectory dependent on that manoeuvre.Despite the fact that this method is not limited by the design of a hand-crafted cost function, data existence, and prediction horizon, the approach does not consider the impact of distance between road users (i.e., closer vehicles have a higher impact on an ego-vehicle trajectory).To deal with this important issue, Dai, et al. [2] proposed a Spatio-Temporal LSTM based model.The advantage of this work compared with the previous work is that this model considers the safe distance between surrounding road users to measure the degree of influence between them.However, to implicitly measure this important influence, a more comprehensive definition of safety must be considered.
Though there have been efforts to develop a variety of trajectory prediction approaches in recent years, they have not taken into account either the associated risk within the traffic scene or the degree of influence of surrounding vehicles.Furthermore, most of the existing approaches are focused on a specific dataset making them impractical for more general application.To investigate this aspect of generality, we evaluated the system across NGSIM and HighD datasets.These datasets were collected in two different countries containing both Change Lane and lane-keeping (Go Straight) trajectories with different speed limits, driving culture and road geometry.The proposed model in this study adopts a different architecture called the Dual Learning Model (DLM) which takes information from two different inputs to predict vehicle trajectory without being specific to one dataset.

III. METHODOLOGY AND MODEL DESIGN
In this section, we describe the problem formulation and our proposed approach.The problem we are addressing can be defined as predicting the trajectory of an ego-vehicle for a certain time horizon while its trajectory is influenced by other road user trajectories.Figure 1 illustrates the proposed framework in this study.The approach illustrated in Figure 1 uses an LSTM encoder-decoder structure for time series prediction allowing an insight into past trajectory values to evaluate how the system will evolve in the future and thus, predict future values.It means that the system remembers important past events and applies this learning to predict future values.We take the encoder-decoder architecture as the base model which is comprised of two sub-models; an encoder to remember important past events, and a decoder to convert the important events into a prediction for the future.
We employ LSTM networks due to their remarkable success in sequence-to-sequence prediction.We use a multimodal information fusion technique to inject relevant features into the encoder-decoder structure by applying a convolutional LSTM mechanism on top of this base model [20].The system extracts the risk features of the traffic scene and models the possible risk interaction among vehicles in the traffic scene on the top of the context vector.Risk is represented by Time-to-Collision between vehicles in the traffic scene.Later, we will show that the proposed model provides a more accurate trajectory prediction result compared with the stateof-the-art.
In the following section, we describe the Encoder LSTM, Convolutional LSTM, Decoder LSTM, and input channels in detail.

A. ENCODER LSTM
The encoder receives multiple input sequences (X ) in the form of trajectory sequences from the ego-vehicle and encodes them to generate the encoded sequences h, which is given by where n is the length of track history, ψ i is the feature vector at each time step i.The feature sequences, ψ i , is obtained from ego-vehicle kinematic information as well as surrounding vehicle information.In Equation 2, x i and y i are lateral and longitudinal coordinates of the ego-vehicle in the i th time step where x 0 = y 0 = 0.The lateral velocity, longitudinal velocity, lateral acceleration, and longitudinal acceleration are shown by v x , v y , a x , and a y , respectively.W v,i , in Equation ( 2), represents weights which reflect the degree of surrounding vehicle influence on the ego-vehicle trajectory.We will justify the use and impact of W v,i on improving the accuracy of trajectory prediction later in this paper.The encoded sequence, h i , is given by an LSTM encoding function, where X i the is input sequence, h i−1 is a hidden state at time i − 1 and c i is the context vector at time i.

B. CONVOLUTIONAL LSTM (ConvLSTM) NETWORK
The aim of this section is to use the salient Risk Map information extracted by the ConvLSTM network.The ConvL-STM structure replaces the inner product of LSTM from the Hadamard product with convolution.It processes hidden and input states in two dimensions making it able to capture spatio-temporal motion patterns.The primary benefit of using this structure is to capture the adjacent spatiotemporal relationships and identify a saliency map of the inter-vehicle interaction based on risk, which can improve the overall accuracy of trajectory prediction.The input is the Risk Map sequence passing through the multiple ConvLSTM layers which have convolutional structures in both input-tostate and state-to-state transitions [21].These structures are represented in the following equations, where σ is the logistic sigmoid function, i t , f t , C t , O t and H t are the input gate, forget gate, cell activation, output gate, and cell output at time t, respectively.' * ' denotes the convolution operator and '•' denotes the Hadamard product.
In Equation ( 6), W represents the filter matrices which connect different gates and b is the corresponding bias vector.
We have added padding before applying the convolution operation to ensure that the states have the same number of rows and same number of columns as the inputs.The ConvLSTM learns to leverage the temporal features during training and compresses the whole input sequence into a hidden state tensor.To obtain the affine transformation parameter, a fully connected layer with a Rectified linear units (ReLu) activation function is used.These parameters are concatenated with the encoder c i and then is fed to the decoder for further processing.The decoder unfolds this hidden state to predict the vehicle trajectory.The context vector contains a representation of input sequences in the encoder and RM which carries the most significant spatio-temporal features from the input sequence.The context vector c dec i is calculated and passed to the decoder as their initial cell states.

C. DECODER LSTM
The decoder module reads the encoded sequences and makes a multi-step prediction in the output sequence.The decoding function, f , takes the context vector, c dec i , at time i, the hidden state and output of the decoder at time i − 1 shown by S i−1 , and y i−1 , respectively, and outputs the predicted trajectory distribution which is given by,

D. INPUT
The proposed framework takes two channels of information as input to the system; trajectory information through the Occupancy Map (OM) and risk associated with the traffic scene through the Risk Map (RM).In the following paragraph, we describe each channel in detail.
The first channel of information is encoder input.The encoder takes track history of the ego-vehicle and all the surrounding vehicles through an Occupancy Map (OM).To define the OM, we surround the ego-vehicle with a stationary grid, where each column corresponds to a single lane, and the rows are separated by a length of approximately one standard vehicle.Vehicles in a traffic scene, adjust their trajectory with the surrounding vehicle trajectories generating interaction amongst them.The impact of this interaction between vehicles is influenced by the relative distance and velocity between them.In this study, we utilise the OM to define the weight vector W v reflecting the impact of surrounding vehicle trajectories on the ego-vehicle.The W v can be calculated by, where d r,i and V r,i are the relative distance and relative velocity between the k th vehicle and the ego-vehicle at the i th time step, respectively.
The second channel of information embedded in the proposed framework is the Risk Map which is an auxiliary input to generate the context vector for the decoder module.This input allows the system to automatically extract the risk pattern associated with the traffic scene in each time step i.In contrast to existing approaches [2] which consider a simplified calculation of the safety factor (i.e., only longitudinal velocity is considered), this paper presents a comprehensive definition of TTC where we consider both lateral and longitudinal velocity to describe the safety factor.The method of how TTC is calculated for both lateral and longitudinal velocity is depicted in Figure 2. Here, (x enc , y enc ) is the encounter point which is where a collision occurs if vehicles continue with the same velocity and direction [22], it is given by: where (x ego , y ego ) and (x k , y k ) are the coordinates of the ego-vehicle and the k th surrounding vehicle in meters, respectively.The θ [Rad] is the heading of the vehicle which is obtained from the angle between lateral velocity, v y , and longitudinal velocity, v x (θ = arctan v y v x ).If vehicle j continues its path with the same heading, t enc,j is the time taken to reach the encounter point.
The TTC between the ego-vehicle and the k th surrounding vehicle can be obtained from t enc,ego − t enc,k .
Here, is a safety margin which defines the degree of hazard for the situation.In = 0, both vehicles reach the encounter point at the same time (i.e., means a collision).The higher value shows that the safety factor is more conservative (i.e., higher value denotes lower risk of collision).The literature shows that a TTC of more than 6 seconds is considered a safe distance between two vehicles [23]; however, we will discuss the impact of TTC values greater than 6 seconds on the accuracy of trajectory prediction in the simulation result section.Figure 3, shows the average of the TTC inverse value in the NGSIM dataset.

IV. SIMULATION RESULTS AND DISCUSSION
The following section describes the datasets, evaluation metrics, and the model performance of the trajectory prediction framework.We utilised LSTM with 128 units for the encoder and decoder module with a batch size of 32.Models are trained using the Adam optimizer [24] with a learning rate of 0.001 and ReLU activation with α = 0.1.The loss function of the training process adopts MSE and we set 20% dropout in the training process.Two ConvLSTM layers are stacked, where each layer has 128 hidden states with 2 × 2 kernels.The decoder consists of the 128 LSTM cell followed by an FC layer.

A. DATASETS AND DATA PREPARATION
To train and evaluate the proposed framework, we first need a detailed dataset covering a variety of vehicle trajectories.We extract all the eligible trajectories from two naturalistic vehicle trajectory benchmark datasets, NGSIM US-101 and I-80 and HighD.The NGSIM US-101 and I-80 dataset [25], [26] consist of real highway driving scenarios captured using multiple overhead cameras observing sections of highway in the US in 2005 at 10 Hz.It is a large size dataset with dense traffic flow which has been widely used in the literature especially for trajectory prediction [2], [4], [11].
The second dataset is the HighD dataset [27] which was captured by a camera-equipped drone in 2017 and 2018 at 25Hz.The dataset is collected from an aerial perspective of six different highways in Germany including 5600 complete lane changing trajectories.
NGSIM and HighD datasets have different proportions of lane changing and straight through trajectories.Table 1, overviews the different trajectory types in both datasets.We randomly divided each of the datasets into training, validating, and testing sets where the training set contains 70% of the trajectories, and the validating and testing sets contain the remaining 10% and 20%, respectively.In each dataset, we consider one of the vehicles as the ego-vehicle and the other vehicles as surrounding vehicles.In the proposed model, we observe 3 seconds of the track history of all vehicles in the traffic scene to predict the next 5 seconds of the ego-vehicle trajectory, therefore each trajectory segment needs to be 8 seconds long.In order to reduce model complexity, we have down sampled the 8 second trajectory with a time step t equal to 0.2 seconds (5 Hz).The down-sampling process provides a sequence with a length of 40 steps.We have utilised the first 15 steps (i.e. 3 seconds) as the track history sequence and the rest of the 25 steps (i.e. 5 seconds) as the output sequence.We also transformed the initial values of each trajectory (x = 0,y = 0) making the model independent of road geometry.The independence of trajectory prediction from road geometry makes the model more flexible allowing it to be employed across different environments.

B. EVALUATION METRICS
We report the trajectory prediction accuracy with the Root of the Mean Square Error (RMSE) metric since it has been widely used in previous work [2], [11], [17], [28].The RMSE is a quadratic scoring rule measuring the average magnitude of the displacement errors of predicted positions and real positions during the prediction time horizon.This metric gives relatively high weight to large errors which penalises them and can be defined as follow, where n is the number of trajectories in the testing set, is the predicted position and x obs i,t is the respective observed positions for the trajectory i at t th time instance.

C. ANALYZING MODEL PERFORMANCE
In this section, we analyse the performance of the proposed trajectory prediction model in both quantitative and qualitative aspects to present the experimental results.
To intuitively show the trajectory prediction outcomes, three sample trajectories including change lane right (first row), change lane left (second row), and going straight (third row) are randomly selected from both the NGSIM and HighD datasets, see Figure 4.The first row shows two examples of lane change where the ego-vehicle performs two types of interaction for the same trajectory type.One scenario with a vehicle in the destination lane and one change lane to a free lane.In Figure 4, the surrounding vehicles' 3 second track history (solid line) and ground truth trajectory (dash line) are shown in red and yellow, the track history (solid line) and ground truth (dash line) trajectory for the ego-vehicle are in navy blue, and the five second predicted trajectory for the ego-vehicle is plotted in light blue.It can be clearly seen that the proposed approach achieved a promising result in predicting different trajectory types while it considered the interaction with surrounding vehicles.

1) WEIGHT VECTOR AND RISK MAP IMPACT
In the experimental results depicted in Figure 5, we investigate the impact of considering a weight vector (W v ) and Risk Map on the accuracy of the proposed system.To evaluate the effectiveness of these factors, we present the RMSE performance of the proposed model for lateral and longitudinal trajectory prediction with three modifications.
In one experiment, we trained the system only with kinematic parameters, in the second, we embed W v as well as kinematic parameters, and finally in the third experiment, we utilised the Risk Map, W v , and kinematic parameters while the other attributes of the three models are unchanged.The goal is to check the accuracy of the trajectory prediction for different time horizons.Figure 5 shows that, in both datasets, systems utilising W v and the Risk Map decrease the RMSE value for both lateral and longitudinal trajectory.It should be noted that this improvement is significantly larger in lateral trajectory prediction showing that W v and the Risk Map provide the system with better lateral trajectory insight compared with solely using Kinematic parameters.In longitudinal trajectory prediction, embedding the Risk Map and W v results in a slight improvement compared to the accuracy of the system which is trained with W v and Kinematic parameters.As an example, in the first row of Figure 5, the RMSE value of the model is decreased by adding the Risk Map (i.e., blue line), however this reduction is more noticeable for lateral trajectory prediction.

2) COMPARISON WITH STATE-OF-THE-ART
To evaluate the proposed approach, we pursue a direct comparison with state-of-the-art vehicle trajectory prediction using the same dataset (i.e., NGSIM).The results show that the proposed method outperforms the state-of-the-art model and decreases the overall RMSE value of the system by 10 percent on average.Table 2, summarises the RMSE values comparing the proposed methods with the baseline trajectory prediction models in the literature [4], [5], [11], [28].The following baselines are evaluated:  The comparison results show that the proposed DLM structure has better performance with ∼11% and ∼14% improvement in RMSE for NGSIM and HighD, respectively.As depicted in Table 2, the RMSE value for the system trained with the NGSIM dataset is higher than the system trained with the HighD dataset.The reason for this difference has also been noted in previous investigations [5] and is likely related to dataset size differences or is due to annotation inaccuracies resulting in physically unrealistic vehicle behaviours in the NGSIM dataset [29].

3) ANALYSING THE ALGORITHM PERFORMANCE WITH INSIGHT INTO THE DATA
Up until this point, we have reported and compared the accuracy of our proposed method with existing approaches.In this part, we aim to analyse the algorithm performance with more insight into the data.Figure 6 shows the plot of RMSE values for different lane changes (left/right) and straight trajectories as well as the average RMSE of the system.The bar chart demonstrates that change lane right and change lane left have a higher RMSE compared with go straight trajectories.
In other words, the system has better performance predicting go straight trajectories than turn trajectories.The available datasets contain a higher proportion of go straight trajectories however most errors happen for lane change trajectories.By reporting the average error of the system, errors in lane change prediction may hidden since the system can accurately predict go straight trajectories.Therefore, to more accurately evaluate and express the representation of model performance, we report the RMSE value of the proposed model for different types of trajectories for a 5 second prediction horizon in both datasets.
Figure 6, shows the prediction error for different trajectories including Change Lane Left (CLL), Go Straight (GS), and Change Lane Right (CLR).It is observed that the error of GS is lower than the other trajectories.This result is expected since lane keeping (Go Straight) trajectories are less complex to predict compared with scenarios where the ego-vehicle intends to change lane.Figure 6 also shows that the RMSE value caused by CLR is more than CLL trajectories indicating the trained system is less confident in predicting CLR trajectories.This can be the result of the lower number of CLR trajectories compared with CLL and GS trajectories existing in datasets.

4) ANALYSING THE INDEPENDENCE OF THE SYSTEM TO A SPECIFIC ENVIRONMENT
To illustrate the independence of the system to a specific environment, we train and test the DLM structure with different combinations of both datasets for training and testing, such as: a) Train and test with the same dataset b) Train with both datasets and test with each of the datasets, separately c) Train with a combinations of both datasets and test with both datasets d) Train with the HighD dataset and test only with the NGSIM dataset (Cross dataset prediction).The result, Figure 7, shows that the system maintains its performance without the tendency to overfit to one of the datasets.Thus, combined training across multiple datasets does not substantially decrease accuracy of prediction within each individual dataset.In the cross dataset experiment, where we trained the system with the HighD dataset and tested it with the NGSIM dataset, the prediction result is less accurate implying that cross learning is not a feature of this approach due to uniqueness of each dataset.However, the training across multiple datasets does not negatively influence the accuracy of individual dataset prediction.In order to see the impact of the Risk Map on the DLM structure, we attempt to gain a better understanding of our model performance in the qualitative analysis.We investigate the effect of the surrounding vehicle's trajectories on the ego-vehicle trajectory prediction with respect to the TTC described in Equations ( 10) and (11).In Figure 8, a trajectory example is presented where the ego-vehicle is interacting with three vehicles in the traffic scene, see Figure 8(a).The ego-vehicle track history starts at the left lane where V 1 is going straight and finishes at the middle lane where there are two vehicles which are going straight.Different TTC values are presented with dot point' colours in the track history of the surrounding vehicles.The TTC pattern changes from |TTC| > 6 yellow scattered points) to smaller TTC values showing the dynamic interaction, and hence risk, between vehicles in the traffic scene, see Figure 8 (b).When the ego-vehicle starts changing lane, the TTC values for V 2 and V 3 become red meaning that risk has increased in that lane.As can be seen in Figure 8 (b), the ego-vehicle begins the lane change manoeuvre but seems to straighten midway, possibly due to the recognition of the trailing vehicle in the new lane.This change in approach is likely due to the driver's re-evaluation of risk as they recognise changes in the vehicle environment.When the ego-vehicle starts to change lane, TTC values for both V 2 and V 3 decrease again (i.e., orange scattered points).The relative risk relationships change due to behaviours of both the ego vehicle and surrounding vehicles, illustrated by differences in levels of risk for V 2 and V 3 over time.Different combinations of behaviours may lead to a similar risk outcome.As equilibrium is achieved between vehicles, risk once again becomes yellow.Lane changing behaviour is influenced by risk recognition by drivers and thus influences the precision of lane change prediction.
Ultimately, the result of this investigation shows that the deep feature selection of the Risk Map using ConvLSTM helps to create an accurate trajectory prediction and captures the dynamic interaction of the ego-vehicle with other vehicles in the traffic scene.

V. CONCLUSION
In this paper, we have presented a novel end-to-end Dual Learning Model (DLM) for vehicle trajectory prediction.We consider all traffic scene interactions including their relative distance and velocity to the ego-vehicle and apply a comprehensive safety factor definition based on TTC which results in precise trajectory prediction.Two channels of information are embedded within the proposed framework.In the second information channel, we used ConvL-STM, wherein the inner product of LSTMs is replaced with convolution to maintain spatio-temporal motion patterns.We added more expressive features to propose a precise trajectory prediction framework without increasing complexity.We investigate the effectiveness of our proposed system by testing and evaluating it using two distinct datasets, NGSIM (i.e.I-80, US-101) and HighD, resulting in ∼11% and ∼14% improvement in RMSE values of prediction for NGSIM and HighD datasets, respectively.Additionally, we would like to point out that this is the first time that the risk associated with a traffic scene is considered in a trajectory prediction framework.It allows the prediction model to learn different patterns of influence from surrounding vehicles which results in more precise trajectory prediction compared with baseline models where they just consider the ego-vehicle and surrounding vehicle kinematics parameters.
We acknowledge the limitation of considering other road environments such as curved roads or roundabouts at this stage of our research.The future direction of this research is to investigate effectiveness of the proposed method for different scenarios such as roundabouts and intersections.In addition, investigating the influence of Cooperative ITS (C-ITS), the effect of using communication information in the proposed dual learning Model for decision making in automated vehicles, is a worth future investigation.

FIGURE 3 .
FIGURE 3. Average of inverse TTC values in NGSIM dataset.

FIGURE 4 .
FIGURE 4. Predicted trajectory using the proposed method for different trajectories including change lane right (first row), change lane left (second row), and going straight (third row).
i. Vanilla LSTM (V-LSTM): This method utilises the track history of the ego-vehicle in the encoder LSTM and produces a unimodal output distribution with the LSTM decoder.ii.Manoeuvre-LSTM (M-LSTM) [11]: This model employs an encoder-decoder framework where the egovehicle and surrounding vehicle trajectories are fed into the encoder.The decoder uses an encoded vector as well as manoeuvre encoding and generates a multi-modal trajectory prediction.

FIGURE 5 .TABLE 2 .
FIGURE 5. Effectiveness of considering W v and Risk Map in the vehicle trajectory prediction.

FIGURE 6 .
FIGURE 6. RMSE value of the proposed model for different trajectories (a) NGSIM, and (b) HighD dataset.

FIGURE 7 .
FIGURE 7. The RMSE value of the proposed system using different training datasets.

FIGURE 8 .
FIGURE 8.An illustration of dynamic vehicle interaction and the effect on risk/TTC.

TABLE 1 .
An overview of different trajectory types in datasets.