A Hybrid Improved Dragonfly Algorithm for Feature Selection

Feature selection, which eliminates irrelevant and redundant features, is one of the most efficient classification methods. However, searching for an optimal subset from the original set is still a challenging problem. This paper proposes a novel feature selection algorithm named hybrid improved dragonfly algorithm (HIDA) which combines the advantages of both mRMR and improved dragonfly algorithm (IDA) in order to generate promising candidate subset and achieve higher classification accuracy rate. Firstly, to generate promising subset, features with small weight have chance to be selected into candidate subset with a small probability in mRMR. Secondly, to balance the exploitation and exploration capabilities of IDA, dynamic swarming factors are proposed to balance global and local capability. Lastly, to enhance the exploitation capability of IDA, quantum local optimum and global optimum are introduced in the position updating mechanism. The performance of HIDA is investigated on ten gene expression datasets and eight UCI data sets from the UCI Machine Learning Data Repository. Results show that the performance of HIDA is superior to BBA, BDA, CDA, LBPSO, MPMDWOA and MSMCCS.


I. INTRODUCTION
Real-life datasets are often characterized by a large number of irrelevant or redundant features. The combination of features requires testing different subsets to find the optimal one [1]. This approach is highly time or space consuming. Mining large data requires the help of machine learning algorithms because of the increasing trend of high-dimensional data collection [2]. Feature selection is able to find a subset of features to decrease the number of features without degrading the prediction accuracy of the classifier [3], [4].
Feature selection algorithms are divided into three categories, filter methods [5], wrapper methods [6], and hybrid methods [7]. Filter methods analyze the internal characteristics of data regardless of machine learning. The advantages of filter methods are low computational cost and suitable for high-dimensional data. The famous Maximum Relevance Minimum Redundancy (mRMR) algorithm [8] and Reli-efF [9] are representative filter methods. Wrapper methods The associate editor coordinating the review of this manuscript and approving it for publication was Wei Xiang . utilize machine learning to get higher classification accuracy with computational overhead.
The meta-heuristic algorithms are very successful machine learning algorithms to solve various optimization problems [10]. From Fig.1, the meta-heuristic algorithms are divided into two families which are called individual-based algorithms and population-based algorithms. The wellknown individual-based algorithms are hill climbing [11], iterated local search [12], and simulated annealing [13]. Population-based algorithms can be classified into three main categories including evolutionary algorithms, swarmbased algorithms, and physical phenomena algorithms. Some of the recently proposed evolutionary algorithms are biogeography-based optimization algorithms [14], evolutionary membrane algorithms [15], and human evolutionary model algorithms [16]. Some of the well-known swarm-based algorithms include Particle Swarm Optimization (PSO) [17], Genetic Algorithm (GA) [18], Ant Colony Optimization (ACO) [19], Bacterial Foraging Optimization (BFO) [20], Artificial Bee Colony (ABC) [21], Cuckoo Search (CS) [22], Whale Optimization Algorithm [23], [24]  and Dragonfly Algorithm (DA) [25]. The DA is regarded as a successful algorithm that outperforms other well-known optimizers due to its simplicity and efficiency [25]. The most recent physical phenomena algorithms in this category are Henry Gas Solubility Optimization [26]- [28], Gravitational Search Algorithm [29], Artificial Chemical Reaction Optimization. Algorithm [30], Ray Optimization [31], Central Force Optimization [32], Kinetic Gas Molecules, and Gases Brownian Motion Optimization [33]. Among these meta-heuristic algorithms, PSO and GA are well-known optimizers. Compared to PSO, high exploitation assists DA algorithm to rapidly converge towards the global optimum. Compared to GA, DA algorithm appropriately balances exploration and exploitation to handle difficulty in challenging search space. Therefore, in this paper, DA is selected as a wrapper method for feature selection.
There are some efficient evolutionary feature selection approaches, such as variable-size cooperative coevolutionary particle swarm optimization algorithm (VS-CCPSO) [34], binary differential evolution with self-learning (MOFS-BDE) [35], two-archive multi-objective artificial bee colony algorithm (TMABC-FS) [36], and return-cost-based binary FFA (Rc-BBFA) [37]. However, VS-CCPSO, MOFS-BDE, TMABC-FS and Rc-BBFA are not suitable for highdimensional datasets. Memory based Hybrid Dragonfly Algorithm (MHDA) introduces internal memory to overcome the premature convergence of DA [38]. However, balancing local and global search capability of DA is not taken into consideration in the MHDA. To accelerate the convergence rate of DA, chaotic maps are employed to adjust five behaviors of dragonfly for high-dimensional data in Chaotic Dragonfly Algorithm (CDA) [39]. However, the chaotic weights of five behaviors are not able to balance exploration and exploitation ability in CDA. To overcome the drawback of premature convergence, a novel hybrid dragonfly algorithm combined with differential evolution (Hybrid DA-DE) for solving global optimization problems is proposed [41]. However, Hybrid DA-DE is not suitable for high-dimensional data.
The Hybrid feature selection algorithm has the advantages of both filter and wrapper algorithm [53]. The hybrid feature selection algorithm has two-stage. In the first stage, the filter algorithm is used to reduce the number of features and remove some irrelevant features. Forming candidate feature subsets is beneficial to improve computational efficiency in the next stage. In the second stage, the candidate feature subsets obtained in the first stage are used as inputs, and the wrapper is used to obtain the desired number of features and higher classification rate. To find the optimal feature sets, some hybrid feature selection algorithms are proposed. Zheng et al. propose a hybrid feature selection algorithm which uses improved mRMR and modified whale optimization algorithm [42]. Zheng et al. present a hybrid feature selection algorithm based on a new filter algorithm and the improved cuckoo search algorithm [43]. Unler et al. come up with a hybrid feature selection algorithm based on particle swarm optimization (PSO) and mutual information [44]. Akadi et al. put up forward a two-stage selection algorithm by MRMR and Genetic Algorithm (GA) [45]. A two-stage feature selection algorithm combining IG (Information Gain) approach and Binary Particle Swarm Optimization (BPSO) is proposed [46]. However, the subsets obtained from the above filter algorithms are always fixed combination and the diversity of population initialization is not rich enough. Thus, it is prone to premature convergence at local optimum in the above algorithms.
To sum up, three shortcomings are not considered in the studies mentioned above. Firstly, the subsets are fixed combination, because low-ranking features that associate with the best fitness have no chance to be randomly selected into candidate subset in mRMR. Hence, mRMR is easy to fall into a local optimum. Secondly, DA utilizes fixed weights to adjust swarming factors, thus the importance of local and global search capabilities is equal in the whole search process. The exploration capability in the early stage and exploitation capability in the later stage are not guaranteed. Therefore, fixed weights are not able to balance local and global optimum capabilities. Lastly, the best fitness of the current dragonfly and the best fitness of dragonflies are not taken into consideration in the position update of DA, so DA fails to guide dragonflies to potential candidate solutions that are associated with the best fitness. Therefore, the exploitation ability of DA is weak.
To address the above problems, we propose a novel feature selection algorithm named hybrid improved dragonfly algorithm (HIDA). Firstly, to avoid falling into a local optimum, features with small weight are randomly selected into candidate subset with a small probability which is decreased with iterations in mRMR. Therefore, the diversity of population initialization of IDA is rich. Secondly, to balance local and global optimum capabilities, dynamic weights which is consist of the separation weight, the alignment weight, the cohesion weight and the enemy weight are decreased with iterations. Four weights adjust the exploration ability of DA. The exploration capability in the early stage and exploitation capability in the later stage are guaranteed by dynamic weights. Lastly, to enhance the exploitation ability of DA, the concept of quantum optimal solution of current dragonfly and optimal solution of the current population are presented. Therefore, potential candidate solutions that are associated with the best fitness have a greater probability to be stored for further exploitation stage.
To sum up, three corresponding improvements are presented in HIDA as follows.
1. The diversity of candidate subset is enriched to avoid falling into a local optimum.
2. To balance global and local capability of HIDA, dynamic weights of swarming factors are used to adjust weights of four behaviors.
3. To enhance the exploitation capability of HIDA, the concept of quantum pbest and gbest are presented.
The rest of our paper is organized as follows: In section 2, we introduce the basic theories of our research. In section 3, we come up with a hybrid improved dragonfly algorithm called HIDA. In section 4. We evaluate the performance obtained with three improvements. In section 5, a conclusion is illustrated in detail.

II. THEORETICAL BACKGROUNDS
In this section, we introduce a filter algorithm and a wrapper algorithm. A filter algorithm is mRMR and a wrapper algorithm is DA.

A. DRAGONFLY ALGORITHM
Dragonfly algorithm (DA) is inspired by the hunting and migration mechanisms of idealized dragonflies. The hunting mechanism is known as static swarm behavior in which the dragonflies search for food sources over a small area by the formation of a small group of dragonflies. The migration mechanism is known as dynamic swarm behavior which is characterized by a massive number of dragonflies flying in one direction over long distances. The static and dynamic swarming behaviors of dragonflies represent the exploration and exploitation phases of meta-heuristic optimization, respectively.
The behaviors of dragonflies follow the principles of separation, alignment, cohesion, attraction towards the food, and distraction from the enemies. To model the swarming behaviors of dragonflies, the mathematical position of DA can be explained as follows.
where i = 1, 2, 3 . . . , N , x d i corresponds to the position of the ith dragonfly in the dth dimension of the search space, D is the number of dimensions and N is the number of search agents.
Separation (S i ) represents the static collision avoidance of individuals from other neighboring search individuals, this behavior is mathematically modeled as Eq. (2).
Alignment (A i ) corresponds to the velocity matching of individuals to other neighboring search individuals, this behavior is mathematically modeled as Eq. (3).
Cohesion (C i ) indicates to the individuals towards the neighboring center of the mass, this behavior is mathematically modeled as Eq. (4).
where X i and V i correspond to the position and velocity of the ith individual, respectively. X refers to the position of the current individual and N denotes the number of neighboring individuals.
In the DA, the fitness and position of food source are supposed to be updated using the best candidate(attraction). In addition, the fitness and position of the enemy should be updated using the worst candidate(distraction). The attraction of the ith individual F i towards the food source is mathematically modeled as Eq. (5). The distraction E i of the ith individual from enemies is mathematically modeled as Eq. (6): X is the position of the current individual, F + represents the position of the food source and E + represents the enemy's position.
DA uses two vectors to update the position of a dragonfly, the step vector ( X ) and the position vector (X ). The step vector represents the movement direction of dragonflies, which is similar to the velocity vector in PSO. The step vector is modeled as Eq. (7): where t is the iteration counter, s is the separation weight, S t i is the separation of the ith individual, a is the alignment weight, A t i is the alignment of ith individual, c indicates the cohesion weight, C t i is the cohesion of the ith individual, f is the food factor, F t i is the food source of the ith individual, e is the enemy factor, E t i is the position of enemy of the ith individual and w is the inertia weight.
If dragonfly has at least one dragonfly in the neighborhood, the position of the dragonfly is updated as Eq. (8). If there is no dragonfly in the neighborhood radius, the position of the dragonfly is updated using Levy Flight equation as given in Eq. (9).
where d is the dimension of position vectors. The Levy flight is calculated in Eq. (10).
where r 1 and r 2 are two random numbers in [0,1], β is a constant (equal to 1.5 in this work), and σ is calculated in VOLUME 8, 2020 Eq. (11).
Peng proposed Maximum relevance minimum redundancy (mRMR) filter method in 2015 [47]. The mRMR method uses Mutual information (MI) which is used to calculate feature correlation and redundancy in two phases.
In the first phase, the MI is applied to calculate the relevancy between the label and the feature. In the second phase, the redundancy between each of the two features is calculated. The mathematical position of mRMR can be explained as follows.
where x i is a feature of feature set X , and D is the number of features.
where s j is a selected feature of X , and M is the number of selected features.
where r m is a remained feature of X , W is the number of remained features and D = M + W . First, the mRMR method calculates the relatancy (Rl) between the label and the feature, with the largest mutual information value of feature being selected.
where I (s i , C) and I (r m , C) are the mutual information value between label (C) and feature.
where C n is the label of X , and H is the number of label. Then, the mRMR method measures the redundancy (Rd) between every two features.
where I (S j , r m ) is the mutual information value between every two features. Finally, the mRMR method selects the feature of maximum relevancy minimum redundancy (mRMR) from the remaining feature sets. I (S j , C) and 1 M is constant, the formula simplified into formula (19).

III. THE PROPOSED ALGORITHM
A. IMPROVED DRAGONFLY ALGORITHM (IDA)

1) QUANTUM PBEST AND GBEST
In DA, all the positions of dragonflies are updated according to Eq. (8) and Eq. (9). This position updating mechanism leads to premature convergence because the dragonflies are not allowed to keep track of previously obtained potential solutions. Therefore, in order to overcome the aforementioned limitation of the DA, pbest and gbest are introduced in IDA. pbest is the best fitness value obtained so far by a dragonfly while gbest is the best fitness value obtained so far by all dragonflies in the neighborhood. Besides, IDA integrates quantum physics to update the position of the dragonflies. During each iteration, the fitness value of a dragonfly is compared with the pbest value in current population. Better fitness value is saved to pbest. The best fitness value obtained so far by all dragonflies is saved to gbest value. A dragonfly updates its position as shown in the following formula (20).
where C 1 and C 2 represents the cognitive and social parameters respectively, C 1 and C 2 are set to 2 by default, P t i and P G represents the best fitness of the i th dragonfly and the best fitness of the swarm upto t th iteration respectively. N is the number of instances.

2) ADAPTIVE PARAMETERS
DA utilizes fixed parameters to adjust swarming factors. Therefore, DA can't balance local and global search capabilities. In order to overcome this shortage, dynamic curves are proposed to tune parameters of swarming factors. In the early optimization stage, IDA explores a huge search space to avoid premature convergence. In the later optimization stage, IDA efficiently exploits small regions to refine the final solutions. Because too many parameters of IDA need to be adjusted, the factors with the same change trend are designed to the same curve distribution. The parameters (the separation weight s, the alignment weight a, the cohesion weight c and the enemy factor e are mathematically modeled in eq. (23). Fig. 2 demonstrates the values of weights vary with iterations. First, the weights are set at a relatively high value and declines slowly to prevent blinding follow-up and falling into local optimum. Then the curve rapidly goes down to slow position in order to balance exploration and exploitation ability. Last, the parameters decrease slowly to enhance exploitation ability.
where t is the number of iterations and Init is the initialization of four parameters. Fig. 3 illustrates the process of IDA. t is the number of iterations and T is the max number of iterations. First, we calculate classification accuracy of position vectors using SVM and update food source, enemy, pbest and gbest. Second, the separation weight s, the alignment weight a, the cohesion weight c and the enemy factor e are decreased adaptively using formula (23). Third, x is updated using all parameters in formula (20). Finally, position vectors are updated based on their classification accuracy rate.

B. HYBRID IMPROVED DRAGONFLY ALGORITHM (HIDA)
The integration of the filter algorithm (mRMR) and the wrapper algorithm (IDA) leads to a Hybrid Improved Dragonfly Algorithm (HIDA). mRMR selects features with maximum relevancy minimum redundancy, the subsets are always fixed combinations. Low-ranking features have no chance to be selected into candidate subset. It has been recognized that the combinations of individually good features do not necessarily lead to good classification performance. Thus, a dynamic number of features are selected for HIDA. From  Fig.4, the x-coordinate is the order of features from mRMR, the y-coordinate is the probability of the feature which will be selected. If the probability is greater than ε, the feature is selected to initialization of IDA. The top-ranking feature has higher probability of being selected. Each dragonfly in the population initialization of IDA is the subset which is obtained from mRMR. The diversity of population ensures that at least one subset in the population is able to contain all interacting features which are associated with label. The  mathematical formulas are formulated in eq. (24) and (25).
where p is the selected probability, x is the feature set which is obtained by mRMR and D is the number of features.
The larger dimensionality of dataset is, the larger ε is. In Fig.5, Every solution is characterized as a single dimensional vector, and the length of the vector is the number of features in the dataset. Every cell of the vector contains two values (1 or 0). Value 1 depicts that the corresponding feature VOLUME 8, 2020  is chosen while value 0 represents that the feature is not selected. Fig. 6 illustrates the process of IDA. First, all variables are initialized. t is the number of iterations, T is the max number of iterations, S is the number of selected features, G is the feature set of selected features, F is the feature set of remained features. Second, we use mRMR to construct a feature subset which has maximum relevancy minimum redundancy. Thirdly, Low-ranking features are randomly selected to a feature subset with small probability using formula (23). Finally, given a subset, IDA is applied to generate a promising candidate subset. The first judgment of ''t < T'' is used to generate candidate subsets for mRMR, and second judgment of ''t < T'' is used to control the number of iterations for IDA.

A. DATASETS DESCRIPTION AND PREPROCESSING
To evaluate the usefulness of the HIDA approach, we carried out experiments on five gene expression datasets [49] and eight UCI datasets from the UCI Machine Learning Data Repository [50], [51]. The characteristics of datasets are summarized in Table 1.

B. PARAMETER SETTINGS
The parameter setting is introduced in Table 2 according to the literatures. The number of particles for five algorithms is set to 50, and the maximum number of iterations is set to 100.
The radial base function (RBF) is used as the kernel function of the SVM model [52]. In the following experiments, cross validation is made up of 10 folds. 9 folds are used for training and the last one is used for testing. All of the codes are implemented in MATLAB R2018b and run on a Window 10 PC with Intel Core i5-3470K 3.20 GHz CPU and 16 GB RAM.
According to the initialization of six parameters of HIDA, four parameters (s, a, c and e are decreased with the iteration number in formula (23) while two parameters (w and f ) are increased with iteration number in formula (26). Furthermore, we present a threshold ε. ε is adjusted depending on the dimensionality of dataset in formula (27). For lowdimension dataset, setting a small value to enrich the diversity of candidate subset. For high-dimension dataset, setting a large value to save running time.
where t is the number of iterations and Init is the initialization of two parameters (w and f ).
where D is the number of features.

C. EXPERIMENT RESULTS
The experiments are tested through implementing BBA, DBA, CDA, LBPSO, MPMDWOA, MSMCCS, and HIDA using MATLAB. Due to the efficiency of SVM algorithm, we adopt SVM in HIDA. Four criteria are used including Acc, average selected features, AUC and F indicator. For balanced dataset, we often use the Acc as the criterion. But Acc is not enough for imbalanced problems. AUC is an indicator used to measure the classification performance in imbalanced data. F indicator is a comprehensive measure for balancing recall and precision. The Acc, AUC and F indicator are computed by the following four formulas.
where TP is the number of positive instances which are classified correctly and FP is the number of negative instances which are misclassified. β is a coefficient which is used to adjust the important degree of recall compared with precision. β is usually set to 1, and it means that recall and precision are important equally.  From Table 3, it is clear that HIDA shows the best performance among all the methods on all datasets except Ozone. HIDA has higher accuracy rate compared to the other methods. Similarly, the average selected features of HIDA is lower than other methods. Thus, the proposed subset generation improves HIDA a lot. Besides, Acc and NumF of IDA outperform BDA on most of the datasets. Take 9-Tumors for instance, Acc and NumF of IDA are 65.67 and 267.8 respectively, while Acc and NumF of BDA are 60.24 and 342.1 respectively. Table 4 shows the Acc of seven algorithms in 18 datasets. Table 5 and Table 6 show and the average number of selected features and max number of selected features respectively. VOLUME 8, 2020  The performance of HIDA is better than the other six methods. In Table 4, it can be observed that Acc of HIDA is superior to other algorithms in all datasets except Ozone Level Detection. The of Ozone Level Detection is lower than the best result while the number of average selected features is superior to MSMCCS which gets the best performance among five algorithms. In Table 5, we can find that MPMDWOA, MSMCCS and HIDA get one, two and fifteen best average selected features separately. In DLBCL dataset, HIDA provides 100% accuracy by using 19 attributes only, at the same time MPMDWOA show 97.5% accuracy and 153.3 attributes. The result shows that HIDA outperforms other algorithms in both classification accuracy and selected attributes for the same dataset.
In addition to Acc and average selected features, Tables 7 and 8 show F 1 and AUC of seven algorithms in 18 datasets. HIDA outperforms all algorithms in 15 and 16  datasets in both F 1 and AUC respectively. Inspecting Table 5, It can be analyzed that HIDA achieves superior F 1 results in fifteen out of eighteen datasets. In Table 8, BDA does not outperform HIDA over most datasets in terms of AUC, whereas it performs better than others on Secom datasets with an insignificant difference. Moreover, it is worth noting that HIDA outperforms other algorithms over fifteen datasets.
The computational complexity of HIDA is O(T * (D 2 + N * D + N * S)). T represents the maximum number of iterations, N represents the number of populations, D represents the number of features in the data set, and S represents the time taken to execute the SVM classifier. In the table, N * D represents the computational complexity of the wrapper algorithm position update in one iteration, N * S represents the computational complexity of the SVM classifier in one iteration, and O(D 2 ) represents the computational complexity of the filter algorithm in one iteration. The computational complexity of MPMDWOA and MSMCCS are the same as HIDA. BBA, BDA, CDA, LBPSO are wrapper algorithms, the computational complexity of four algorithms is O(T * (N * D + N * S)). Therefore, the computational complexity of the hybrid algorithm is higher than the wrapper algorithm.
Additionally, in order to better validate the superior convergence behavior of HIDA on test datasets, Fig.7 is provided to demonstrate the convergence behavior of five algorithms on 18 datasets. By directly drawing curves of classification accuracy rate with the iterative number, we can see the classification accuracy rate is increased monotonously  in each iteration until level off. As it can be observed from these figures that HIDA converges fast compared to other algorithms and achieves best classification accuracy rate in most datasets. The results demonstrate the strong exploitation ability of HIDA in the later stage of optimization.
To further assess HIDA, boxplots for classification accuracies are obtained in comparison with the other methods. These plots are shown in Fig.8. The plots show that HIDA is better than the rest of the methods considered in most of the cases. Table 9 shows that six pairs of Wilcoxon signed-rank tests are made on Ozone data set. With the significant level 0.05, it can be detected the performance of HIDA is statistically significant compared to BBA, BDA, CDA, LBPSO, MPMDWOA and MSMCCS. VOLUME 8, 2020 D. DISCUSSION According to the above observations, it can be concluded that the performance of HIDA is superior to the other algorithms for feature selection. The main reason for the good performance of HIDA is the integration of mRMR and IDA. On the one hand, IDA is able to enhance the exploitation and exploration capabilities. One the other hand, the proposed subset generation enriches population diversity and helps HIDA jump out of local optimum.
Firstly, Acc and NumF of IDA outperform BDA on most of the datasets in Table 3. IDA has higher accuracy rate compared to the other methods, because adaptive quantum optimum enhances the exploitation capability of IDA. Secondly, due to high accuracy and minimized feature size selected of HIDA in Table 4 and Table 5, we can conclude that HIDA significantly enhances the exploration of DA. Lastly, by analyzing the results of F 1 and AUC in Table 7 and Table 8, HIDA outperforms all algorithms in 15 and 16 datasets in both F 1 and AUC respectively. We can say that random selection plays a complementary role in enhancing exploration in the HIDA algorithm besides adaptive quantum optimum which enhances exploitation. Moreover, F 1 and AUC indicators are appropriate to measure imbalanced classification results, thus HIDA has an excellent performance in resolving imbalanced classification problems.
Despite efficient results generated by HIDA, yet it maintains certain limitations because of the nature of the hybrid algorithm whose computational complexity is higher than the wrapper algorithm. Moreover, mRMR sequentially constructs the feature subset by including one feature at a time. As a result, the mutual information of a feature is calculated and used to weigh the feature based on the feature subset available. Therefore, there is a dependence on the feature subset construction sequence in mRMR, and mRMR is not able to achieve high classification accuracy rate.

V. CONCLUSION
Due to the low classification accuracy rate of DA, a novel HIDA optimization algorithm which is combined with mRMR is proposed for feature selection. Firstly, a novel subset generation with random selection improves the classification accuracy rate of HIDA. Secondly, each dragonfly is designed to keep track of its coordinates with the addition of internal memory in quantum space. The novel position updating mechanism is able to balance local and global search capabilities of HIDA. Lastly, to enhance the exploitation capability of DA, HIDA utilizes dynamic parameters to adjust swarming factors.
To demonstrate the effectiveness of HIDA, experiments have been conducted on 8 UCI and 10 microarray datasets. Compared with the other six algorithms, HIDA is able to escape from local optima. Although HIDA doesn't have the best performance in all datasets, the five criteria of HIDA are better than other algorithms in all microarray datasets. HIDA is more suitable for the high dimensional dataset. The results verify that the integration of IDA and mRMR improves HIDA significantly. YUEFENG ZHENG received the Ph.D. degree in computer application from the Jilin University, in 2019. He is currently with Jilin Normal University. His research interests include feature selection and object detection. VOLUME 8, 2020