Robust Vehicle Classification Based on Deep Features Learning

This paper aims to introduce a scientific Semi-Supervised Fuzzy C-Mean (SSFCM) clustering approach for passenger cars classification based on the feature learning technique. The proposed method is able to classify passenger vehicles in the micro, small, middle, upper middle, large and luxury classes. The performance of the algorithm is analyzed and compared with an unsupervised fuzzy C-means (FCM) clustering algorithm and Swiss expert classification dataset. Experiment results demonstrate that the classification of SSFCM algorithm has better correlation with expert classification than traditional unsupervised algorithm. These results exhibit that SSFCM can reduce the sensitivity of FCM to the initial cluster centroids with the help of labeled instances. Furthermore, SSFCM results in improved classification performance by using the resampling technique to deal with the multi-class imbalanced problem and eliminate the irrelevant and redundant features.


I. INTRODUCTION
During the last decades there have been major changes in passenger vehicle sizes around the world. One of the main functionalities of these changes is global competition, which forces car manufacturers worldwide to quickly improve and introduce competing vehicles. The implication is significantly changes in the vehicle segments (increased share of SUVs), vehicle dimensions (increased size of the vehicles within each segment), and other design parameters like powertrain type and power. To date, numerous vision-based vehicle classification methods are available. However, illumination changes, shadows, partial detections, occlusion, and camera viewpoint changes have strong impact on these techniques [1]- [6]. This complexity further increases for vehicles that belong to different classes despite having similar dimensions or have visually similar appearance but dissimilar dimensions [7]. Furthermore, the review of related literatures [8]- [11] shows that the growth of car models over time poses an additional challenge to the accurate vehicle classification based on dimensions. In fact, the most popular names of car models have remained intact since they were developed, The associate editor coordinating the review of this manuscript and approving it for publication was Shun-Feng Su .
making it difficult to follow the evolution of sizes during vehicle classification (Table 1). In this setting, Fig. 1 illustrates the core challenge of vision-based vehicle classification which is high intra-class (within the cluster of classes) variation and relatively low inter-class (between classes of multi-class classification) variation [12]- [14].
This issue reveals the need of using automated and efficient classification techniques for different vehicle types for a variety of applications. For a vehicle classification system to be useful in real-world conditions, it must be robust to inter-class interferences in the data by considering new measurement method based on the discriminative features.
One approach that has received much attention in recent years is semi-supervised learning (SSL) techniques. SSL is a branch of machine learning that aims to classify large size of the unlabeled data using the minimal size of labeled information set to build better learning [15]- [19]. SSL are particularly relevant to scenarios where the input labeled set being able to utilize the classifier during the training process and then applied out-of-sample approaches to make predictions on unlabeled data. In recent years, SSL approaches attempt to improve the performance in sequential learning of supervised and unsupervised learning by utilizing information generally associated with the other and repeating the   process until convergence [20]- [22]. SSL algorithms can be classified into the Supervised Deep Learning using Pseudo Labels [23], Spatio-spectral LapSVM [24], Semi-Supervised Ladder Network [25], Temporal Ensembling with augmentation [26], Mean Teacher, weight-averaged consisting targets [27] and semi-supervised fuzzy rough convolution neural network (FRCNN) [28]. Most of these algorithms often consider a balanced training set which may increase the overlapping between majority and minority class instances. This paper is an extension of previous work originally focused on the preprocessing data technique by feature extraction to classify data into the five different categories as part of a multi-class vehicle classification system based on both technical and geometric parameters using fuzzy and non-fuzzy clustering methods [7]. However, the classification performance metrics was found to be slightly affected with the imbalanced problem. The problem of imbalanced dataset appears in most traditional classification methods when the proportion of majority class instances across the known classes is biased or skewed, i.e. the number of instances in a majority class is much higher than the number of instances in a minority class [29]- [31].
In recent literatures many different types of algorithms and techniques are proposed to deal with the problem of class imbalance datasets, such as random sampling, learning algorithms, and feature selection [32]- [38]. Inspired by these methods, we develop a new semi supervised approach which utilizes both supervised and unsupervised data to alleviate the multi-class imbalanced problem, eliminating the redundant features and also exploiting hidden information during the clustering process.
In this paper, we propose a novel Semi-Supervised Fuzzy C-Mean (SSFCM) clustering approach based on the feature learning technique, which is a highly useful technique for representation learning with high dimensional datasets containing high-level of uncertainties. In our approach, Fuzzy C-Mean (FCM) clustering use to predict labels for unsupervised data. For classification, we suggest that one can use unsupervised and supervised data along with the predicted labels to extract the discriminative information for classification. Following this, in order to enhance the prediction ability, we combine feature reduction techniques (feature extraction technique and feature selection technique using Random oversampling (ROS)) to handle the multi-class imbalanced problem and eliminate the irrelevant and redundant features for improving the classification accuracy. To evaluate the performance of these algorithms we use both technical and geometric parameters, and apply them to a vehicle fleet dataset that includes all new registered passenger cars in Switzerland.
This work focuses mainly on the performance of vehicle multi-class classification and vehicle intra-class classification. The aim of our investigation is to accurately analyze vehicle changes enabling automated vehicle classification of large databases. The experimental results demonstrate that the performance metrics of proposed SSFCM approach are on average better than those of other tested methods. Furthermore, the potential of proposed approach with ROS in improving the imbalance multi-class classification performance on vehicle dataset increases the quality of clustering results over FCM.
From the experimental validation, our model can achieve acceptable accuracy (∼86%) using only 10% of the labels rate for each class. All the experiments demonstrate that the proposed method is feasible and effective in categorizing vehicles based on their features.
The rest of the paper is structure as follows. Section 2 briefly summarizes the main results of related research works. In section 3 we present a detailed description concise details of the datasets, algorithms and experiments. Lastly, in section 4 we present the majors findings of this work and provide recommendations for further research.

II. RELATED WORK
A large number of works have applied machine learning techniques for the classification of vehicles, including unsupervised (clustering) and supervised (classification) methods [16], [23], [24], [26], [27], [39]- [67]. Among the unsupervised techniques, FCM clustering algorithm is the most popular. Javadi et al. [42] applied FCM with dimension and speed features to classify vehicles into ''private car'', ''light trailer'', ''lorry or bus'' and ''heavy trailer'', reaching an accuracy of 96.5% on a dataset composed of 400 vehicle images. Yao et al. [43] developed an axle-based vehicle detection and classification method using FCM to identify and segment vehicle axle pixels from camera images. With this vision-based approach, they reached a vehicle detection rate of 62.8%. Saraçoğlu and Nemati [44] used FCM for image segmentation based on the vehicle dimensions combined with Support Vector Machine classification to classify the vehicles as ''small vehicle'', ''big vehicle'' or ''others''. Researchers have also carried out multiple comparisons of FCM with other methods, which demonstrated that the FCM algorithm is capable to overcome some of the problems faced with noise sensitivity defect and non-linear data clustering [45]- [47]. Velmurugun and Santhanam [48] compared the clustering performance and effectiveness of the K-means (KM), K-Medoids, and FCM clustering algorithms using different shapes to cluster arbitrarily distributed data and finding mutual exclusion clusters. Moreover, Joyti and Kumar [49] compared the performance of KM and FCM algorithms in terms of computational time, while Gosh and Dubey [50] computed the performance and clustering accuracy of these two methods based on the efficiency of the clustering output as well as the computational time.
In contrast, other works have applied supervised methods for the classification of vehicles. Zhang et al. [51] developed a length-based method for vehicle detection and classification, reaching an accuracy of 97% for truck classification. Arunkumar et al. [52] used a neural network classifier based on geometrical features and appearance-based attributes to classify passenger vehicles into brands. Moussa [53] also used geometric-based and appearance-based features for multi-class (''small'', ''medium'', and ''large size'') and intra-class (''pickup'', ''sport utility vehicle'', and ''van'') vehicle classification using a support-vector network mod-el. Lastly, Cheung et al. [41] presented a vehicle classification method based on measurements with magnetic sensors to classify the vehicles into six types (''passenger vehicle'', ''SUV'', ''van'', ''bus'', ''mini-trucks'', and ''truck''). Their algorithm achieved an accuracy of 80% to 90% when vehicle length was used as a feature, compared to only 60% when the length was not considered.
These works highlight the potential of using machine learning techniques for feature-based vehicle classification. However, in most cases they separate vehicles in classes that differ greatly in appearance and size (e.g. mopeds, passenger cars, vans trucks, buses), while separating more alike subclasses within these categories (e.g. the passenger car subclasses) poses a greater challenge, especially for vision-based methods, and requires expert knowledge.
Lately, feature learning techniques have showed outstanding performance for addressing uncertainty problem for clustering and classification [24], [54]- [59]. The classification performance highly depends on the quality of features generated from the data as input to the classifier process. However, only a limited number of studies have been done on combine feature learning techniques to improve classification performance on the high dimensional and multi-class imbalanced datasets. The classification method proposed in this paper is a new semi-supervised clustering scheme SSFCM that incorporates semi-supervised information in FCM algorithm to considerably improve its effectiveness [22], [16], [23], [26], [27], [60]- [63]. More details about the feature learning techniques can be found in the article by Jiang et al. [64], in which they combined several feature extraction methods with a support vector machine classifier to group the vehicles in six categories, namely ''large bus'', ''passenger car'', ''motorcycle'', ''minibus'', ''truck'' and ''van''. This study VOLUME 9, 2021 achieved a classification accuracy of 97.4%. Balid et al. [65] implemented a deep learning-based classification model that uses the vehicle length as a key feature and classifies vehicles into ''passenger vehicles'', ''single unit trucks'', ''combination trucks'', and ''multi-trailer trucks'' with a classification accuracy of 97%. Maungmai and Nuthong [66] used a convolutional neural network method to classify the vehicle type as ''small'', ''medium'', ''large'', and ''unknown'', and vehicle color as ''black'', ''blue'', ''white'', ''green'', ''yellow'', ''red'', and ''unknown''. The convolutional neural network classifier achieved an accuracy of 84.65% for the vehicle type and 75.44% for vehicle color classification. The results comparison with utilized decision trees, random forest, and densely deep neural network classifier show that the classification of vehicle type accuracy increased by 1.8%, and vehicle color increased by 0.8%. Dong et al. [67] proposed a vehicle type classification method using a semi-supervised convolutional neural network from high-resolution vehicle frontal view images. The algorithm achieved 88.11% accuracy.

A. SEMI-SUPERVISED CLUSTERING
Semi-supervised learning falls somewhere between unsupervised learning and supervised learning, which combines both labeled and unlabeled data. It solves classification problems, such as the imbalance of data categories, by using supervised learning algorithm that helps boost the classification accuracy. There are three main types of semi-supervised learning: semi-supervised classification [56], [68]- [71], semi-supervised dimension reduction [24], [72] and semi-supervised clustering [25], [73]- [75].
Semi-supervised classification focuses on minimizing the square loss of predicted errors in the labeled examples, while part of instances is unlabeled; semi-supervised dimensionality reduction attempts to reduce the variance of classifiers in high dimensional data or find the low-dimensional space; semi-supervised clustering aims to increase the accuracy of the defined clusters by identifying better clusters than the ones obtained from unlabeled data [19], [75]- [79]. Typically, semi-supervised clustering methods obtains worse representation results in the original feature space. To make the semi-supervised clustering more efficient, it is reasonable to do semi-supervised clustering with deep feature learning [62], [63], [16], [80]. The framework of the proposed clustering approach is depicted in Fig. 2.
Unlike the most widely used approaches, in SSFCM clustering based feature extraction technique, we consider the three types of information (diffusion label, extract core data, and ex-tract feature vectors) in order to improving accuracy of classification and decrease class imbalance and multiclass overlapping problems. This framework includes two main layers. The first layer contains the labeled (supervised) dataset which is split into train set to create a classifier based on the core dataset and test set to evaluate its output. The second layer contains the combination of recordings from train set not used for creating the classifier along with the unlabeled dataset and provides the input for feature learning process. The cluster centroids that are evaluated in feature learning process are used as an input for extracting feature vectors. Then feature selection process is implemented to eliminate irrelevant and redundant features. Lastly, a SSFCM model is constructed based on the selected feature vectors of the train set and is validated using predicted labels of test set.

B. SEMI-SUPERVISED FUZZY C-MEAN CLUSTERING
Fuzzy C-means (FCM), as an overlapping clustering algorithm, is one of the most popular fuzzy clustering meth-Algorithm 1 Fuzzy C-Means Membership and Centroid Input: Data X whose number of elements N , A, C, m, max. iteration number (T ), error threshold (ε) Output: u ki , v i Set t = 0 1. Initialize v i 2. Update t = t + 1 3. Compute u ki 4. Compute v i 5. If t > T or ||u t − u t−1 || < ε then stop; otherwise 6. Repeat from step 3.
ods [81]. This technique is a soft clustering algorithm. By this we mean that each data point has a probability of belonging to each cluster with partial membership values ranged from 0 to 1. However, due to the non-convexity of its objective function, it may fall into local optimal solution during optimization. To address this issue, we propose semi supervised fuzzy C-means clustering (SSFCM) algorithm that incorporates deep feature learning in FCM to further improve its effectiveness and eliminate redundant information [22], [28], [75]. This method aims to minimize the objective function (J) as follows: where N is number of data elements, C is the number of clusters; X k represents the data k of X = {X 1 , X 2 , X 3 , . . . , X N } in the i th cluster; u ki represents the weighted squared errors function known as membership function; m is a weighting exponent that determines the degree of fuzziness; A is a positive and symmetric (n × n) weight matrix; U is the membership degree of data elements X into c cluster; v i is vectors of center in i th cluster; K denotes the features, and x k − v i 2 A denotes to the Euclidean distance function and it is computed in the A norm between j th data and i th cluster center.
The SSFCM algorithm comprises of the following steps: After calculating deep FCM membership degrees and centroids using the algorithm 2, we select the features (s ⊂ K) using random oversampling (ROS) and Euclidean distance metric techniques. The purpose of ROS approach is to maintain a balance between the features subsets of labeled classes and unlabeled data elements [22], [38], [82]. The Euclidean distances is the most applied (dis)similarity or distance metric Algorithm 2 The Training Strategies for Semi-Supervised Fuzzy C-Means Input: N data elements X = {X 1 , X 2 , . . . , X N }, C, K , labeled dataset (L), unlabeled dataset (UN), membership degree (U ), max. iteration number (T ), error threshold (ε) Output: u iL , u UNL , v iL , v UNL Set t = 0 1. Initialize v iL , v UL 2. Update t = t + 1 3. Compute u iL , u UNL 4. Compute v iL , v UNL 5. If t > T or ||J t − J t-1 || < ε is fulfilled for all labeled and unlabeled objective functions separately then stop; otherwise 6. Repeat from step 3. to measure the similarity between the labeled and unlabeled feature vectors. The outcome is the maximum average of the maximum relevant and minimum redundant features between the each selected feature of unlabeled data and labeled classes [17].

Algorithm 3 Semi Supervised Fuzzy C-Means Classifier
Two methods, micro-averaging and macro-averaging, are used to evaluate the accuracy and performance of classification [22], [83]- [85]. In selecting the best approach for evaluating the performance of a given classifier is important to consider the class-balance and expected outcomes.
A macro-average tend to estimate the metric for each class separately and then compute the overall average. In contrast, the micro-average aggregates the contributions of all classes to estimate the average metric. One specific performance evaluation allows assessing a classifier based on the particular perspective and frequently fails to measure others. Therefore, in this study the macro-averaged accuracy (MAcc) and micro-averaged F-Measure (MFM) are used for the validation of our approach. The MAcc can be defined as: Precision and recall are defined as follows: The MFM is then computed as: Here, Pr denotes precision in macro-average method; TP i (True Positives) is the proportion of the data points classified correctly to class i; FP i (False Positives) is the proportion of the data points that do not belong to class i but are classified to class i incorrectly; FN i (False Negatives) is proportion of the data points that are not classified to class i but which actually belong to class i; n is the number of data points; and c is total number of classes.

IV. EXPERIMENTS A. DATA PREPARATION
In experiments, the vehicle database containing annually new registered passenger cars, vehicle technical specifications and vehicle expert classification data obtained from Swiss Motor Vehicle Information System [86], Federal Office Technical Information [87] and Vehicles Expert Partner [88] respectively. In the preprocessing step we filter the database in order to extract vehicles that do not meet the definition of typical passenger cars, such as small pickup trucks, standard pickup trucks, vans, special purpose vehicles (SPVs), sports cars and multi-purpose vehicles (MPVs) [7]. By considering the goal of this paper, the dataset is separated into two parts, training part and testing part. Training dataset contains 275,601 passenger cars registered in Switzerland in 2018 along with 22 parameters including geometric parameters such as height, length, width, wheelbase, angles of the vehicle front and rear, etc., and technical parameters such as power, engine capacity, energy efficiency, drivetrain, etc.
In the first step of learning process the training dataset is considered to contain two types of patterns: unsupervised (unlabeled) and supervised (labeled) dataset. For this purpose, we used labeled dataset from the traditional unsupervised algorithm, where the FCM clustering algorithm is used in determining passenger cars multi-class classification based on its features [7]. Fig. 3 demonstrates the benchmark of Swiss vehicle datasets in 2018 that shows brief properties of multi-class imbalanced datasets. The total 366 unique samples is grouped into six classes: micro class containing 18 samples, small class containing 50 samples, middle class containing 110 samples, upper middle class containing 84 samples, large class containing 60 samples and luxury class containing 44 samples, where the average imbalance ratio is 3. The average accuracy rate of unsupervised FCM clustering algorithm in compare to the Swiss expert classification was approximately 79% where the optimal number of unsupervised FCM clusters is equal to five [7].
Due to the some limitations of the unsupervised FCM clustering algorithm, the accuracy of features based clustering is not very high. The main reason is that the feature of misclassified samples has a low degree of category membership. Therefore, in this study we used the labeled data with a membership degree greater than 0.95 as the core dataset to extract the accurate classification of misclassified samples and provide the basic for the later step of training. The labeled data is divided into train set and test set. In feature learning process, the unlabeled data and the previous labeled along with the labels of the core dataset are used as input. Prior to model development, new features are extracted to reduce the number of features. In feature extraction step, the cluster centroids learnt by using algorithm 2 and each patch is transformed to a feature vector. In feature selection step,  resampling (ROS) technique is used in order to increase the number of extracted features from minority groups and make it equal to the number of features in majority groups. Finally, algorithm 3 is used to select the best features based on the Euclidean distance and remove redundancy from the feature vector. After we initialized all parts, pseudo labels (PL) of labeled data are assigned to unlabeled data. Following this, all training data with their pseudo labels is carried out to pre-train a SSFCM classification algorithm according to the extracted discriminative features. Finally, new SSFCM is constructed using only the labeled data with true labels.
To evaluate the performance of the proposed approach, we used 10, 20, and 30 percent of the labeled samples from each class to show the effect of the number of labeled data and the rest are used as unlabeled samples. As essential input arguments of unsupervised FCM, the maximum number of iterations was set to 100, the convergence value to ε = 1e-6 and the degree of fuzziness to m = 2. The classification performance of the SSFCM with different rates of labeled samples on the imbalanced multi-class benchmark dataset is presented in Table 2 and compared to the performance of the unsupervised FCM.
The experiment results show that the use of semisupervised learning in fuzzy clustering algorithm can significantly enhance the classification accuracy. Since in the traditional FCM algorithm the initial cluster centers are selected randomly, the algorithm can be easily trapped in local minimum. In contrast, in our approach, the seed training set which contains some labeled data determines the initial centers. As a result, we get considerably better performances (Fig. 4). When the rate of labeled data for each category increases, the center in seed training set is more nearer to the center in dataset. Hence, the accuracy rate slightly improves along with the labeled rate, which remains at about 91%.
The classification accuracy results in Table 3 show that the proposed SSFCM method achieved in most cases the best performance on the passenger vehicle dataset for all labeled rates (only DES-ML with a labeled rate of 10% is slightly higher). The average accuracy over all labeled rates is highest for the SSFCM, showing that the proposed approach outperforms the other semi-supervised approaches.

B. PERFORMANCE MEASURE
To verify the underlying assumption that feature extraction leads to improve classification results compared to the initial classifier's predictions, we use two performance measures to investigate the classification performance of the traditional unsupervised FCM with the original features and SSFCM algorithm, namely MAcc and MFM. Table 4 shows that, compared with the FCM-based classifier, the performance of the SSFCM with only 10% labeled rates is better for all vehicle classes. On average over all classes, SSFCM reaches a MAcc of 85.58% and a MFM of 0.85, which is higher than that of traditional unsupervised FCM. Furthermore, the visual comparison of the performance metrics resulting from the unsupervised FCM and the SSFCM algorithms reported in Fig. 5 (MAcc) and Fig. 6 (MFM), illustrates the superior performance of the SSFCM in confront to the traditional FCM.
The better results of SSFCM approach show the positive impact of taking the class imbalance problem into account for the higher classification accuracy. Therefore, the   experimental results demonstrate that the SSFCM algorithm can extract richer information from vehicle dataset and obtain higher discriminative recognition rates than FCM-based classifier does. This might be due to the variety in geometry between the classes that can be trained effectively with the feature extractor. Our proposed approach can, not only effectively address the problem of multi-class imbalance data but also improve the classification performance.
Lastly, we carried out our experiment to assess the performance of our proposed vehicle classification approach for intra-class vehicle classifications, specifically for the separation of sport utility vehicle (SUV) from non-SUVs. The analysis of drivetrain technologies for each segment based on the expert classification demonstrates that more than two third of the SUVs are four-wheel drive (4WD), while most of the non-SUVs designs are front-wheel drive (FWD) and rear-wheel drive (RWD) [88]. Therefore, the drivetrain technologies taken into account as an assumption of labeled dataset for more accurate vehicle classifications, particularly for big data analysis. Out of the total 366 unique samples only 156 samples meet 4WD assumption. To evaluate the performance, the supervised labeled dataset was split into two subsets (SUV and non-SUVs passenger cars) along with their geometrical and technical features. According to the dataset evaluation SUV (SUV1) contains 96 samples and non-SUVs (SUV0) contains 270 samples, respectively. Fig. 7 demonstrates multi-class and intra-class vehicle classification based on the SSFCM approach. Vehicles are categorized into six main classes (micro, small, middle, upper middle, large and luxury classes) and two sub-classes (SUV and non SUVs). The performance of the SSFCM with only 10% labeled rates for the intra-class vehicle classification reaches a MAcc of 87% and a MFM of 0.91.

V. CONCLUSION
To summarize, we designed a novel SSFCM algorithm for passenger vehicle classification, and compared an initial evaluation thereof with an expert segmentation to investigate higher levels of vehicle classification. To incorporate the unlabeled data in the training of SSFCM, we used their predicted labels generated by FCM clustering algorithm. In this way we generate more useful features for classification and it also helps to effectively solve the multi-class imbalance problem.
Our SSFCM algorithm first uses a feature extraction technique and then clusters dataset based on the maximum average of the maximum similarity between the selected features. In experiments with the new registered passenger car dataset, the SSFCM technique was significantly more accurate than other tested unsupervised and supervised approaches. As a result, the average of both evaluated performance measures (MAcc and MFM) are the highest for our proposed approach.
The proposed approach enables accurate automated vehicle classification of large databases, which in turn facilitates the analysis of fleet changes. Moreover, our automated approach has several important advantages over the expert-based segmentations, such as reduced classification costs and training times, and elimination of subjectivity factors that often hinder the comparison of vehicle classification databases across the world. A further area of potentially fruitful research would be to investigate better ways for generating supervised labels that are more consistent to the underlying semi-supervised labels, as the quality of the supervised labels has an effective impact on the quality of the features extraction and the final classification performance of semi supervised deep fuzzy approaches. The analyze time complexity of targeted resampling technique will further help in investigating the performances of the resampling techniques on multi-class problems.