Deep Transfer Learning Based Intersection Trajectory Movement Classification for Big Connected Vehicle Data

Trajectory movement labelling is an important pre-stage for predicting connected vehicle (CV) movement at intersections. Drivers’ movement prediction and warning at intersections ensure advanced transportation safety and researchers use machine learning-based data-driven approaches to implement these technologies. However, prediction of drivers’ movements at intersections requires labelling the train and test dataset accurately with different vehicle movements at intersections to evaluate the performance of the prediction model by comparing the actual and predicted intersection movements. Moreover, due to GPS detection error or missing co-operative awareness messages (CAM), the data resides with many abnormal trajectories which are unable to be matched with regular straight or any turning movements. Especially for big data with million trajectories, it is tedious to label the movements manually. To solve this problem, we have created an automated trajectory movement classification technique using a dual approach of map matching technique and deep transfer learning modelling. Data of connected vehicle trajectory information is taken from the Ipswich Connected Vehicle Pilot (ICVP) Project, which is one of the largest connected vehicle pilots within a naturalistic driving environment in Australia. Map matching approach is performed as initial labelling by analysing the origin and destination of the vehicle CAM messages at intersections and then was converted as image datasets of 19202 samples. The map matching error and abnormal trajectories are identified by visual inspection. With properly labelled 9496 training images, 10 transfer learning models are built and tested through the remaining 9706 testing images. The maximum testing accuracy (99.73%) is achieved from the Densenet169 model, and the result shows satisfactory accuracy for individual classes: straight (99.85%), turn left (99.59), turn right (99.25), u-turn (100%), abnormal (98.63%). This model becomes a routine tool that is used daily to automatically classify thousands of trajectory movements of the C-ITS data in the ICVP project.


I. INTRODUCTION
The connected vehicle is an emerging technology of Intelligent Transportation System (ITS) that has potential road safety applications and advanced transportation facilities [1]. It is an advanced transformative solution to traffic collisions, vulnerable road users' safety, road work zone safety, congestion, and dilemma zone problems [2,3]. Here, vehicles and system infrastructures are connected wirelessly with a vehicular ad hoc network (VANET) and communicate with each other to properly navigate with upcoming circumstances. The communication transmits bidirectionally from vehicle to vehicle (V2V) and vehicle to infrastructure (V2I) in order to locate vehicle positions, intended movements, destination and produce safety warnings based on circumstances [2]. The multipurpose implementation of this technology and consequent safety warning for drivers improve transportation in many ways [4,5]. Some most frequent safety applications of connected vehicle technology include back of queue warning [6], road work warning [7], advanced red light warning [8], red-light running prediction [3], emergency vehicle travel time reduction [9]. In Queensland, Australia, vulnerable road users: pedestrians, bicyclists, and motorcyclist fatality is 12.3%, 3% and 13% respectively of overall road death [10].
Connected vehicle technology is effective in producing warnings for vulnerable road users crossing at roads. Apart from safety implications, connected vehicle technology ensure emission reduction and fuel-saving which is environmentally friendly as future mobility [11].
At present, there exist many pilot studies of connected vehicles in developed countries like the USA, Australia, Japan, China, France and so on. Some common and renowned connected vehicle pilots are New York City DOT Pilot [12], Tampa-Hillsborough Expressway Authority Pilot [13], Wyoming DOT Pilot [7], Ipswich Connected vehicle Pilot [14,15], Safe and Intelligent Mobility Project [16] etc. They are working on implementing connected vehicle technology and applications practically on-road as field operation test (FOT) over large urban areas. Such largescale pilot studies confront major challenges with accurate data collections, management, processing and analysis.
When evaluating drivers' behaviour under the influence of C-ITS, identifying and labelling individual trajectory movements of connected vehicles at intersections is essential for accurate analysis and understanding. Labelling can be a labour-intensive task that also introduces some level of error. An automated intersection trajectory movement classification is the desired approach for handling big connected-vehicle datasets. As an alternative, there are other commercial routing tools that extract accurate geo-location and map information during dynamic vehicle movements. However, such commercial tools incur a license fee, and this expense rises immensely with usage over very large datasets. Moreover, the quality of collected data highly relies on the transmission latency of information between vehicles and infrastructures. During the data collection in such extensive field operation test procedure, errors are occurred by communication devices like GPS and cause inaccurate CAM location information, which raises the difficulty in properly classifying vehicle trajectories. Especially at intersections, this inaccurate location information leads to obscurity in labelling the vehicle trajectory movements. An example of an erroneous trajectory found during data analysis is shown in the following Figure 1. These vehicle trajectory movements are inconsistent, mismatched, or incompatible with regular straight and turning movements at intersections. So, it is also a desired concern to classify these abnormal trajectories for the purpose of tracking error, defining error reasoning statements and data cleaning. Otherwise, the downstream analysis of the FOT data will not be conclusive with the mislabeled or erroneous dataset. For further use of the data in various applications such as prediction of driver movement or red-light running incidents at intersections, the training of the machine learning model needs accurate labelling of drivers intended movements at the intersection. Otherwise, the trained model's performance will not be reliable as the training and testing data labels are noisy. Even if the trained model performance appears satisfactory, it may produce unacceptable false alarms if used on roads and may even distract drivers and compromise road safety. Thus, accurate and cost-effective trajectory labelling for data analysis purposes and building prediction models is extremely important. Geographic Information Systems (GIS) and vehicle kinematic information like speed, acceleration, braking and other forces related to diving behaviour from naturalistic driving dataset was used to extract driving patterns. They also analyse the effective parameters of GIS mapping of naturalistic driving data. However, no study is found on automated intersection movement scenarios analysis with connected vehicle data using the deep transfer learning technique [17,18].
In this study, an automated trajectory movement classification technique is developed using a dual approach of map matching technique and deep transfer learning modelling for big, connected vehicle data. Initially, a Shallow Map Matching approach is used to label trajectory movements at intersections. The performance of map matching based trajectory movement labelling is unsatisfactory with many mislabelled movements, and this approach is incapable of identifying abnormal trajectories. Further, the trajectory events are converted into images to generate an image dataset of 19202 samples. The mislabeled images are corrected by manual visual inspection, and the images of incomplete, missing and erroneous trajectories are labelled as abnormal Trajectories. At the final stage, this image dataset is used for the Deep Transfer Learning technique, which is considered more flexible than building a deep neural network (DNN) as DNN requires much bigger labelled data to get high performance which is excessively repetitive and timeconsuming. Instead, 10 pre-trained models are trained for accurate classification of the five intersection movements, including abnormal trajectories, and the best model is found with a satisfactory and reliable test result. This procedure helps classifying trajectory movement accurately (near 100%), including abnormal trajectories, and it does not bear any expense like commercial tools. This model has potential application for connected vehicle pilot studies to: • Identify erroneous abnormal trajectories which have missing CAM information due to GPS error at intersections.

•
Automatically annotate trajectory movement for downstream statistical analysis or as for any supervised machine learning based prediction analysis like drivers' intended intersection movement prediction.

•
Create models suitable for hardware capacity. Moreover, this study compares many pre-trained networks of different sizes to annotate trajectories at intersections so that hardwires with different computational power can use the best performing model based on its capacity.

II. WORKING PRINCIPLE
Our working principle comprises several stages. For our current study, naturalistic driving data of connected vehicles is taken from the ICVP project [14], the largest connected vehicle pilot study in Australia. The naturalistic driving method provides intuition to regular driving behaviour, and it is advantageous in collecting very large datasets in quantitative and qualitative terms from field operational test (FOT) [17][18][19]. The ICVP project has installed roadside units at 29 signalised intersections and undergoes a field operation test of 351 connected vehicles on Queensland roads. The map matching approach is performed using road topology information from MAP Extended Messages (MAPEM) and polygon drawing over desired intersection zone using Google Earth Pro software. By analysing the origin and destination of the CAM messages at these polygon regions, vehicle trajectory movements at intersections are labelled initially. Following that, trajectory information is converted as image datasets of 19202 samples using our MATLAB app. The map matching error and abnormal trajectories are identified by visual inspection over the images, and they are labelled into five classes, including straight, turn left, turn right, U-turn and abnormal trajectory class. By the end of the map matching and visual inspection, we take a data set that consists of 19202 images with pure labels. Consequently, 9496 images are used to train 10 pre-trained models through transfer learning, and the rest are used to test the trained models. The flowchart in Figure 2 shows the Automated trajectory movement labelling framework. The detailed schematic procedure of trajectory movement identification is illustrated in the following subsections.

III. MAP MATCHING BASED MANEUVER LABELLING
This section describes the first step in labelling vehicle trajectories at the signalised intersections. This step matches the trajectory points to the surveyed lane centre lines to identify the origin and destination of a trajectory in the intersection segments. The surveyed lane centre lines are extracted from the MAPEM. The steps to do this labelling are as follow:

A. PREPARING SCENARIO
Initially, a set of polygons are constructed for the 29 signalised intersections. Google Earth Pro is used to draw the polygons, which are then saved as a KML file. Finally, the KML is imported into MATLAB and converted to a structure. Figure 3 shows an example of the polygon drawing approach over the intersection.
Preparing pure small dataset Transfer learning VOLUME XX, 2017 We import the trajectory data points and use the polygon structure to select all CAMs inside the intersection area. Then we filter out all CAMs which has a speed value of less than 4.32 km/h. We consider any CAM with a speed value of less than 4.32 km/h is broadcasted by a stationary vehicle. The removal of stationary CAMs can minimise duplicate data points to improve data processing efficiency.

B. ORIGIN-DESTINATION (OD) APPROACH
We use the constructed Origin-Destination (OD) matrix of the intersection to estimate the origin and destination of the trajectories. Because of inaccuracies in the location information, we group the lanes of each leg into two groups, namely the ingress group and the egress group, as shown in Figure 3. Each ingress and egress of intersections is assigned with a unique id. Based on a schematic analysis of origin and destination, we can assign the label of trajectory movement. Segmentation of a four-leg intersection into numeric ids based on ingress (labelled with red fonts) and egress (labelled with green fonts) is shown in Figure 4, and trajectory labelling scheme with origin and destination is shown in Table I.

C. MATCHING TRAJECTORIES WITH OD MATRIX
We calculate the distance from each point on the trajectory to the intersection centre point and use these distances to divide the trajectory into two parts ingress trajectory and egress trajectory. Then we match the constructed ingress trajectory CAM points against all possible ingress lane centre lines extracted from the intersection's KML file. Remove any CAM that is far more than five meters from the nearest centre line as they are deemed to add noise to the data matching. Based on this match, each ingress lane has a score reflecting its probability of being part of the ingress trajectory. We repeat the above procedure using the egress trajectory and the egress centre lanes. After that we use the output lane scores from ingress and egress matching and the approach OD matrix to assign a label (i.e. straight, turning left, turning right and U-turn) to the trajectory. The flowchart in Figure 5 depicts the overall map matching procedure.
Using the map machine based automated labelling, four types of connected vehicle trajectory movements are labelled at a signalised intersection consisting of straight, turn left, turn right and U-turn movements. However, there are no ground truth labels of trajectory movements to compare our map matching based labelling, and so, it is not feasible to measure the accuracy of map matching based labelling. Moreover, our map matching approach is incapable of identifying abnormal and erroneous trajectories that do not match with regular four intersection movements. The trajectories with missing CAM messages are not identifiable using this algorithm as it only analyses the origin and destination of vehicle trajectory points. So, we converted intersection trajectories into images using our designed MATLAB application and inspected their movements visually to compare with map matching based labels. Vehicles positions at trajectory are continuous dots that originate from frequent CAM message-based communication. These continuous dots generate the trajectory line at images which define its movement at intersections with its origin to destination approach. Also, using the vehicle position information at longitude and latitude in the map, the trajectories of each event are drawn into images. The graphical user interface (GUI) of the MATLAB app is shown in Figure 6. To evaluate the result of map matching based trajectory movement labelling, we visually inspect all 19000 trajectories. Here, many incorrect labelling is detected when visually inspecting the trajectory images, and the estimated accuracy is 90%. Also, the abnormal trajectories were identified during the visual inspection of images which were unable to be detected by the map matching approach. So, only labelling with map matching may produce further mislabeled data of intersection trajectories with a lot of noisy abnormal movement incidents. These abnormal trajectory movements found through GPS tracking are often caused by data incompleteness, double trajectory lines or undefined movement class. It is noted that some trajectory origins have displaced a few steps ahead due to missing a few CAM messages, but they are still clearly identifiable of their movement class. So, instead of considering them as abnormal trajectories, they are still considered under regular movement classes. Also, sometimes vehicles parked or passed through a sideway after crossing the intersection. These incidents still show incomplete trajectories, but their movement is still identifiable. But trajectory movements that are not identifiable to the regular four-movement class are considered abnormal and don't contribute to our research purpose. Instead, these can be considered as an error in data collection or detection equipment (GPS). These abnormal trajectories are needed to be identified such that any downstream analysis of the data becomes more accurate. So, it is crucial to identify five different trajectory movement classes, including abnormal trajectories. Figure  7 and Figure 8 shows output labels and some abnormal trajectories, respectively. The assigned label in Figure 7 shows the mistaken label.  To classify trajectory movements with a higher level of accuracy, we used deep transfer learning-based modelling. The detailed schematic procedure of trajectory movement identification is illustrated in the following subsections.

IV. DEEP TRANSFER LEARNING-BASED MANEUVER LABELLING
Transfer learning is a machine learning design methodology where a model developed for a task is reused as the starting point for a model on a second related task. It is a popular methodology in deep learning where it is not feasible to label millions of data points to learn the massive number of parameters in a neural network. Therefore, pretrained models are used as the starting point on a second related task. In the context of our trajectory classification problem, transfer learning uses knowledge learned from the image classification task for which millions of labelled data is available in identifying the trajectory movement type where only a little labelled data is available. Recall that creating labelled data is time-consuming and expensive, and our goal is to reduce human efforts and cost in trajectory classification.
In general, there are two common transfer learning approaches we can use in our trajectory classification problem, namely: develop a model approach and a pretrained model approach. In the developed model approach, we start by selecting a source task related to our task. This source task must have massive data to train a deep neural network from scratch. Then, we use the source task data to train the deep neural network and ensure that it learns the source task. Finally, we reuse and finetune this model or parts of it in our trajectory classification task. The pretrained model approach is different from the above approach in that we directly reuse and finetune one of the models trained on large and challenging datasets and released by many research institutions. We adopted ten pre-trained deep neural networks of different sizes to classify trajectories. The rationale of using ten models is to identify the best performing models for accurate classification of trajectory movements. Moreover, complex networks are often beyond the limit of hardware capacity and computational power. So, it is crucial to show the prediction performance of different models so that the best performing model based on hardware capacity can be chosen for the users' benefit. These models were trained using a massive number of very high-quality images to classify images into one of many predefined categories. The selected pretrained model used in this formula are described in following TABLE II.   TABLE II  In the transfer learning technique, the pre-trained models discussed in Table II are finetuned. All the layers are kept frozen, and only the fully connected layer is changed based on the number of movement classes of our current study. To create deep transfer learning models, we needed a labelled dataset with a very large number of intersection movements. The ground truth labels are critical to justify the model accuracy and performance. So, at first, we pick a portion of ICVP data with 19202 trajectory movement incidents which is initially labelled using a map matching approach. Then we convert them into image datasets and correct the mislabeled images by inspecting them visually. Also, we create the abnormal trajectory movement class as the fifth class in the image dataset so that we can identify them separately. Considerably, 19202 trajectory movements generated 19202 images of trajectory movements. Among them, a training dataset is created with 9496 images, and 10 pre-trained models were built for classifying the trajectory movements accurately. The remaining 9706 images are considered as the test dataset. Figure 9 shows examples of 5 different class data used for training and evaluation.  First, we proceed with image augmentation with random flip, random rotation (0-360 degree) and input shear from -10 to 10 to increase the variety of datasets. This adds more noises in training data and helps the model understanding better situation assessment. We also up-sample the abnormal trajectory and U-turn images in the training dataset. Gaussian noises are also augmented with U-turn images and up-sampled. After up-sampling, the image dataset for all classes during training and testing is illustrated in the following TABLE III. Also, we convert the image size to model specific input size: 224X224 for all pre-trained models except 227X227 for Alexnet. For trajectory movement classification, the training and testing of pre-trained models were performed in python using Pytorch library. It was run on high configured PC with intel i7 10th generation 3.6GHz processor, 16 GB RAM. We use a GPU Nvidia GeForce RTX 3060 Ti which have 8GB GPU memory and 3584 CUDA cores, to help in faster parallelisation of our classification accuracy by classes. We split the training data into 90% for training and 10% for validation. As hyperparameter tuning, we tune the batch size, optimiser and learning rate using Random hyperparameter search tuning with optuna python library which we have found very flexible to integrate with Pytorch library. As an optimiser, we use Adam optimiser for all models except Shufflenet shows better performance in stochastic gradient descent (SGD) with a momentum of 0.9 Learning rate 0.0001 as default works well for all models, and we run the training and validation for around 25 epochs with early stopping criteria on best validation accuracy. After tuning, the selected hyperparameters for trained models are shown in following TABLE IV.

V. RESULT & DISCUSSION
After training the pre-trained models with training images datasets, they are tested with the remaining test dataset of 9706 sample images. The overall model accuracy is measured as well as the prediction accuracy of individual movement classes. The accuracy of all ten models are shown in following Table V, including their test accuracy for each individual intersection trajectory movement class. Right class, which is slightly higher (0.25%). Also, Googlenet performs best in classifying Straight movements, but the performance difference between Googlenet and Densenet169 in identifying the straight movement is very narrow. Among the other models, Vgg16 shows comparatively superior performance classifying all movement classes.  Figure 10 shows Densenet169 as the best performing model with the highest overall prediction accuracy and reliability for all movement classes. However, comparing models based on accuracy is not sufficient to justify the model performance. For the variation of test dataset size for different classes, confusion metrics, precision, recall, f1score is required to understand the performance of the model in real-time implementation. The Confusion Metrics of the best four models is illustrated in TABLE VI  Table  VI, Densenet169, Alexnet and Googlenet models correctly classify the highest number of turn left (724), turn right (792), and straight (8088) movements, respectively. The correct classification rate of the abnormal trajectory (72) and u-turn movement (11) is equivalent to the best four models. However, overall, Densenet169 and VGG16 are reliable with the accurate classification of all movements. Alexnet and Googlenet models classify turn right and straight movement in the highest number, but their performance is less reliable for classifying other classes. Instead, Densenet169 and VGG16 models have correct classification rates for all movements. For further evaluation of model performance, the performance metrics of the best four models are shown in Table VII. ). The comparison of performance metrics among the best four models is visualised in the following Figure 11 along with the precision, recall and the f-1 score of models for individual movement classes. Based on the performance metrics comparison of the best four models, DenseNet169 is considered the best performing model for our current study. Overall, Dense169 classifies the trajectory movement for accurate labelling of mislabelled data and identifying abnormal movements.
In this study, a limited number of U-turn samples and abnormal trajectories are tested, evaluating model performance as the dataset was collected from naturalistic driving and field operational tests. Also, the study classifies abnormal trajectories which only found in the field operation test of the ICVP project. So, model performance may vary if abnormal trajectories of different categories are required to be tested except found in the ICVP projects. However, the methodology of this study is beneficial in solving such limitations. If the model is trained with different categories of erroneous or abnormal trajectories found in a particular experiment, it can successfully classify any intersection movements and abnormal trajectories. This model has successfully classified over 1000,000 trajectory movements of the C-ITS intersection data collected by the ICVP project. It removed a substantial amount of manpower that would have been required to classify the movements with a high level of accuracy. Although we are using transfer learning the model, it still needs a significant number (around 19000) of manually labelled trajectories for training the model. Recent advanced AI technologies like using Generative Adversarial Network may help in producing similar trajectory images of multiple classes and reduce the burden of manual data labelling for supervised learning techniques [27]. In our proposed study, the four best models are explained, and they can be used for different applications. For any cases of memory shortage in testing phases, a smaller network that supports the hardware can be chosen. The methodology of this study has future potential for automated data labelling and scenario analysis of various domains using artificial intelligence techniques. Especially, our prediction models can be directly used labelling big data for supervised machine learning techniques and filtering erroneous data in predicting drivers' turning behaviour. Moreover, the proposed methodology is useful for automated data labelling applications or classifying scenarios of industrial purposes and field operation tests in a large scale.

VI. IMPLICATION
The proposed methodology defines a simple approach of labelling trajectory movements at intersections. It is highly recommendable for connected vehicle pilot studies with its accurate classification performance. The selected model is currently being used in our ICVP project. Any co-operative transport assistance and prediction at an intersection require accurate ground truth labelling to measure the accuracy between actual vs predicted values. Especially for turning movement prediction at intersections, red-light running behaviour prediction or stop-go prediction at amber light requires accurate labelling of vehicle movement at intersections. In the case of a big, connected vehicle dataset, a small accuracy loss may cause mislabelling of huge vehicle movement events at intersections. Even a good prediction accuracy will not be considered reliable when the ground truth is not properly labelled. This will jeopardise the quality of evaluation for the implementation of connected vehicle technology on roads. Our proposed framework is highly accurate (near 100%) for the practical implementation of connected vehicle pilot studies. In most case scenarios, commercial tool like Google direction API is potentially used to identify the vehicle trajectory movement information through geolocation and maps. However, the ICVP project would like to develop our own intersection trajectory movement labelling tools so that we do not need to rely on any commercial product. Moreover, Google direction API has its expenses in license sharing and peruse on data; and this expense is noticeable for large datasets of connected vehicle pilot studies. Our proposed methodology is free of cost and ensures accurate trajectory movement classification. For research purposes at pre-processing data stage, abnormal trajectories and missing information in data are needed to be disregarded as data cleaning [28]. Identifying abnormal trajectories due to CAM message error is crucial to potential data analysis and understanding. Our proposed classification model also helps to identify abnormal trajectories which are created due to CAM message error and unable to be matched with regular straight or any turning movements.

VII. CONCLUSION
This research demonstrates a dual methodological approach to classify vehicle trajectory movement at intersections. It is capable of handling big, connected vehicle datasets of a pilot study by proposing an automated approach of trajectory movement labelling using a simple map matching algorithm and deep transfer learning. The proposed methodology is also cost-effective rather than using expensive commercial tools for identifying vehicle geo-location dynamically and trajectory movement labelling. Also, it helps in data cleaning problems and dataset error identification. The accurate prediction rate of our proposed model in this research defines the potential to use this methodology in the real-time application for trajectory movement labelling of big, connected vehicle data

ACKNOWLEDGMENT
Profound acknowledgement goes to the Department of Transport and Main Roads (Queensland), Australia, for providing the data of our current study. This research is funded by the Department of Transport and Main Roads, the Queensland University of Technology, iMOVE Australia, and supported by the Cooperative Research Centres program, an Australian Government initiative.