Novel Meta-heuristic Algorithm for Feature Selection, Unconstrained Functions and Engineering Problems

This paper proposes a Sine Cosine hybrid optimization algorithm with Modified Whale Optimization Algorithm (SCMWOA). The goal is to leverage the strengths of WOA and SCA to solve problems with continuous and binary decision variables. The SCMWOA algorithm is first tested on nineteen datasets from the UCI Machine Learning Repository with different number attributes, instances, and classes for feature selection. It is then employed to solve several benchmark functions and classical engineering case studies. The SCMWOA algorithm is applied for solving constrained optimization problems. The two tested examples are the welded beam design and the tension/compression spring design. The results emphasize that the SCMWOA algorithm outperforms several comparative optimization algorithms and provides better accuracy compared to other algorithms. The statistical analysis tests, including one-way analysis of variance (ANOVA) and Wilcoxon’s rank-sum, confirm that the SCMWOA algorithm performs better.


I. INTRODUCTION
Stochastic algorithms are traditionally characterized as a heuristic, although current research often labels them metaheuristic. According to Glover's example, all-natural algorithms are termed metaheuristic [1]. Generally speaking, heuristic refers to the process of finding or detecting through trial and error. Meta-shows a level achieved above and beyond the fundamental heuristic as they are not problemspecific. In his foundational work [1], Fred Glover coined the word "metaheuristics" as "a master technique that drives other heuristics towards the local optimism to generate answers that have to be produced differently" [2]. So, metaheuristics can be considered as randomized local searches. While quality solutions may be found to optimize problems within an acceptable amount of time, there is no guarantee that the optimal solution might be achieved. It is most probable that these techniques will succeed. Yet, this is impossible. For high probability global optimization, almost any metaheuristic technique may be used [3].
Meta-heuristics have two characteristics in terms of search behavior: intensification and diversification [4]. Diversifying entails developing various solutions that look at the search field across the globe and intensifying implies restricting the search field to a limited area with superior information. A proper balance between intensity and diversity should be maintained throughout the solution selection process to speed algorithm concordance. The solution is selected for optimum convergence while randomization enhances the search for the location of Optima. A balance between these two components often offers worldwide optimism [5], [6].
Optimization is the process of finding a solution to a given optimization problem that provides the greatest or least objective function value. It is the subject of a wide variety of machine techniques that draw on artificial neural networks [7], [8]. Hundreds of famous optimization algorithms have become accessible, and dozens of technologies are available in major scientific code libraries. Given the problems of optimization, selecting what methods can thus be challenging for solving such problems. Optimization is how a function has the lowest or highest output of input parameters or arguments to be optimized. In machines where the input parameters of the functions like the floating-point values are numerical, continuous functions optimization frequently arises. The function returns an evaluation of the parameter of real life. Continuous function optimization may be used to distinguish between such problems with discrete variables, known as combined optimization problems [9].
Different techniques may be resolved, organized, and called to optimize the problems involving continuous functions [10], [11]. The information about the objective function utilized and used throughout the optimization process depends on one technique of optimization classification. The more information about the target function is generally understood, the easier it is to optimize because knowledge can be applied effectively. Perhaps the significant difference between optimization techniques is identifying the destination function in one location. It means that the first derivative of the feature may be used to identify a possible solution (gradient or route). It distinguishes itself from other notcalculated gradient data [12].
Metaheuristic optimization is the optimization process utilizing metaheuristic techniques. Almost every area of life is involved, much from engineering to business, holiday preparation to internet travel and other applications [13]- [15]. The use of those readily available resources must be maximized due to the continuous scarcity of money, resources, and time. The vast majority are non-linear, multimodal, and restrictive in real-life problems. Different objectives frequently collide. Even if one goal is set, optimal solutions are not always available. Usually, a faultless or failed response is not simple to find. A range of popular metaheuristic algorithms is covered in this article [16], [17]. This paper introduces a hybrid Sine Cosine (SC) Modified Whale Optimization Algorithm (WOA) called SCMWOA. Although the WOA algorithm shows superiority in various single-objective optimization problems, it suffers from local optima stagnation and a low convergence rate. The WOA is considered simple, capable, flexible, and easy to be utilized and the distinctive advantages of WOA cannot be achieved using traditional optimizers. By increasing the number of random agents in the modified WOA (MWOA), the global search can be more effective and be achieved to avoid local optima. The SCMWOA algorithm is proposed by balancing the updating process of the agents' positions in the search space between the sine cosine and the modified WOA algorithms during iterations to avoid the low convergence rate.
The next scenario examines the SCMWOA algorithm's ability to solve benchmark functions divided into unimodal and multimodal functions. Twenty-three functions are employed in this scenario. The SCMWOA in the second scenario is compared to original GWO [18], PSO [19], WOA [21], Feedforward Error Propagation (FEP) algorithm [29], Gravitational Search Algorithm (GSA) [30], GA [25] algorithms, Enhanced Grey Wolf Optimizer (EGWO) [31], hybrid of Crow Search Algorithm (CSA) and GWO (GWO-CSA) [31], and hybrid of SCA and PSO (bSCA-PSO) [28]. The third and last scenario is designed in this work for testing the ability of the algorithm for solving classical constrained optimization problems of tension/compression spring design (TCSD) [32] and welded beam design [33]. In addition, the SCMWOA algorithm results are compared in the third scenario with the original GWO [18], PSO [19], WOA [21], and GSA [34] algorithms' results to get the minimum cost.
The main contribution of this work can be summarized as follows.
• A Sine Cosine Modified Whale Optimization Algorithm (SCMWOA) is presented. • A binary SCMWOA is presented. • Ability of the binary SCMWOA algorithm in feature selection problems is tested. • Ability of the SCMWOA algorithm to solve twentythree benchmark functions is tested. • Ability of the SCMWOA algorithm for solving two constrained optimization problems of Tension/Compression Spring and Welded Beam designs is confirmed.
The following sections are organized as follows. The materials and methods of WOA, modified WOA, and SCA are discussed in Section II. Section III and IV present the proposed SCMWOA algorithm in continuous and discrete forms. Section V shows the results and discussion of the designed scenarios of feature selection, benchmark functions, and solving constrained optimization problems. Conclusion and future work are introduced in Section VI.

II. MATERIALS AND METHODS
In this section, the WOA, the modified WOA, and the SCA optimization algorithms are presented.

A. WHALE OPTIMIZATION ALGORITHM
This algorithm was first proposed in 2016 [21]. It mimics the bubble-net foraging strategy of humpack whales. In this algorithm, a number of n whales in the WOA algorithm can "swim" in an n-dimensional search space. To get the food (global solution), the position of each whale should be updated in the space search during iterations. To achieve this, the following equation was implemented in the WOA algorithm.
where the vector X(t) represents the t th iteration's solution.
The vector X * (t) indicates the prey's possible position. The "." symbol between vectors represents the pairwise multiplication. The A and C vectors are updated during iterations as A = 2a.r 1 − a, C = 2.r 2 . a is decreasing linearly from 2 to 0. r 1 and r 2 are selected randomly between [0, 1].
The exploitation phase of the WOA algorithm is based on a shrinking encircling mechanism that decreases with the value of a, and a spiral updating and is calculated as the distance between whale's location and location of the prey. The process of spiral is expressed as in the following equation.
where l is selected randomly between [−1, 1]. The spiral's shape is represented by the constant b. To simulate the process of prey's encircling and spiral movement, this equation is applied.
where r 3 is selected randomly in [0, 1] to control switching between a spiral or circular movement. On the other side, the exploration Phase (searching for a prey) is done based on the vector A. By this process, the agent goes away from the leader. Thus, the agent position will be updated according to a random whale X r . This allows the optimizer a more global search. This can be achieved by the following equation.
The A vector is used to control switching between exploration and exploitation. The termination criterion of the WOA algorithm will be due to the number of iterations. The pseudo-code of the original WOA algorithm is shown in Algorithm 1.

19:
Calculate objective function F n for each X i

B. MODIFIED WHALE OPTIMIZATION ALGORITHM
As presented in the original WOA algorithm in the previous section, the position of search agent is changed/updated based on only one random whale, named X r , that is determined from the population randomly to give the optimizer a more global search capability (exploration ability). By increasing the number of random agents in the modified WOA (MWOA), the global search can be more effective and be achieved. The following equation is applied to replace equation 4 of the original WOA algorithm for increasing the number of random agents up to three agents and give the algorithm more exploration ability.
where the three random agents are indicated as X α , X β , and X γ which are employed in the MWOA algorithm instead of one random agent. The parameter of ζ is computed as follows.
where M ax iter as maximum number of iterations during the execution process. The w 1 , w 2 , and w 3 parameters are selected randomly in [0, 1]. VOLUME 4, 2016 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Initialize SCMWOA algorithm population X i (i = 1, 2, ..., n), size n, maximum iterations M ax iter , objective function F n . 2: Initialize SCMWOA algorithm parameters a, A, C, l, r 1 , r 2 , r 3 , r 4 , r 5 , r 6 , r 7 , w 1 , w 2 , w 3 , t = 1 3: Calculate Objective function values F n for each agent X i 4: Find best solution X * based on F n 5: while t ≤ M ax iter do 6: for (i = 1 : i < n + 1) do 7: if (t%2 == 0) then 8: if (r 3 < 0.5) then 9: if (|A| < 1) then 10: Update current agents' positions based on the following equation Select three different random agents X α , X β , and X γ from the population 13: Update ζ by the following equation. Update current agents' positions based on the following equation using random agents Update current agents' positions based on the following equation  if (r 7 < 0.5) then 21: Update current agents' positions based on the following equation X(t) + r 4 × sin(r 5 ) × |r 6 X * (t) − X(t)| Update parameters a, A, C, l, r 3 , r 7

28:
Calculate objective function F n for each agent X i and update old values 29: Find best solution X * based on F n and update old value 30: Set t = t + 1 31: end while 32: Return best solution X *

C. SINE COSINE ALGORITHM
The SCA algorithm was presented in [35] by switching between the sine and cosine based functions. To know the direction of the movement and how far the movement will be, SCA is based on a set of random variables. The following equation was used to update positions in this optimizer.
where the position of current solution id represented as X(t), while the best solution is indicated as X * (t). The r 5 , r 6 , and r 7 parameters are selected randomly in [0, 1] during iterations. To make a balance between the process of exploration and the process of exploitation, r 4 is changed during iterations as follows.
where t as current iteration, a as constant, and M ax iter represents the maximum number of iterations. Apply the SCMWOA algorithm 8: Update the solutions to binary solutions by the following equation Sigmoid(X * ) = 1 1 + e −10(X * −0.5)

9:
Evaluate objective function for each agent 10: Update parameters 11: Update the best solution based on the objective function 12: end while 13: Return the optimal solution

III. PROPOSED SCMWOA META-HEURISTIC ALGORITHM
The presented Sine Cosine Modified Whale Optimization (SCMWOA) algorithm is explained in this section. The SCMWOA algorithm is shown in Algorithm 2. The presented algorithm is proposed by balancing the updating process of the agents' positions between the sine cosine and the modified WOA algorithms during iterations.
The SCMWOA algorithm starts by initializing the population and algorithm parameters, then calculates all agents' objective functions to get the initial best solution in steps from 1 to 4. For iterations t%2 = 0 of current iteration t, the modified WOA algorithm is applied in steps from 8 to 18. As presented in the modified WOA algorithm, the position of the search agent is changed and updated based on three different random whales, named X α , X β , and X γ , that can be determined from the population to give the algorithm more global search capability. While for the rest of the iterations, the cosine algorithm is applied in steps from 20 to 25. Steps from 28 to 30 are applied to update the parameters and find the current best solution X * .
The SCMWOA algorithm' computational complexity according to Algorithm (2) will be discussed. Let n as number of population; M t as total number of iterations. For each part of the algorithm, the time complexity can be defined as: • Updating parameters a, A, C, l, r 3 , Considering the number of variables as m, the final computational complexity of the algorithm will be O (M t × n × m).

IV. BINARY SCMWOA ALGORITHM
The SCMWOA algorithm has a binary version based on MWOA and SCA. To get a probability value of two discrete classes, an activation function, named Sigmoid, can be employed for binary classification [36]. The classification using this function gives output values between zero or one. The optimizer's outputs are changed to binary values from the continuous ones by the following equation.
where the best position is represented by X * , The function Sigmoid is mainly scales the continuous values to 0 or 1 values and is calculated as follows.
The proposed binary SCMWOA algorithm is discussed step by step in Algorithm 3. The binary SCMWOA algorithm starts by initializing the population and algorithm parameters in step 1. All the solutions are changed to binary ones in step 2. The algorithm calculates the agents' objective function values to get the initial best solution in steps 3 and 4. The k-NN model training and the error are calculated to adjust the performance at step 5. The SCMWOA algorithm is then applied in step 7, and the output solutions are updated to VOLUME 4, 2016 binary ones in step 8. Steps from 9 to 11 evaluate the objective function redetermining the best solution and update the algorithm parameters for the next iteration.

V. EXPERIMENTAL RESULTS AND DISCUSSION
The experimental results section is divided into three scenarios. The first scenario is designed to test the ability of the SCMWOA algorithm in feature selection problems based on nineteen different tested datasets from the UCI public machine learning repository. The next scenario examines the presented algorithm's ability to solve twenty-three benchmark functions divided into unimodal and multimodal functions. The third and last scenario is designed in this work for testing the ability of the algorithm for solving two constrained optimization problems of Tension/Compression Spring and Welded Beam designs.

A. FEATURE SELECTION SCENARIO
Nineteen UCI repository datasets are tested in this work to analyze the ability of the proposed algorithm for feature selection problems. The nineteen datasets, shown in Table  1, are determined with various number of features/attributes, instances, and classed that the algorithms may be evaluated on working with various concerns. In this scenario, the presented algorithm of binary SCMWOA (bSCMWOA) is compared to the original bGWO [18], bPSO [19], bSFS [20], bWOA [21], bMVO [22], bSBO [23], bFA [24], bGA [25] algorithms, Modified GWO (bMGWO) [26], hybrid of PSO and GWO (bGWO-PSO) [27], hybrid of GA and GWO (bGWO-GA) [26], and hybrid of SCA and PSO (bSCA-PSO) [28] in which b denotes the binary variant of the algorithm. Configuration of the presented SCMWOA and compared algorithms during the experiments are discussed in Tables 2  and 3 with 100 iterations and 10 agents initiated at the start of each algorithm.
For evaluating the feature selection ability of the proposed algorithm, the following metrics are employed in experiments. For M as the number repetitions, g * represents the optimal solution, and N be the total number of points. The following equation can compute the Average Error for L as a class for point, C as classifier output for that point, and M atch to represent the matching between the two inputs.
The Average Fitness can be computed, for size(g * j ) as the vector g * j size and D represents the size of dataset, as follows.
The results of the presented and compared algorithms based on average error for the nineteen datasets are shown in Table 4. Average selected size-based results are presented in Table 5. The average fitness, best fitness, and worst fitnessbased evaluation results are introduced in Tables 6, 7, and 8, respectively. The standard deviation fitness results of the tested algorithms are shown in Table 9. Tables 10 presented the p-values of the proposed and other tested algorithms for the nineteen datasets, which reflects the performance of the suggested algorithm with a p-value less than 0.005 for all datasets. Figure 1 shows the feature selection average results acquired over all the datasets, summary results, to measure the performance of the bSCMWOA algorithm. The results  shown in tables from Table 4 to Table 10 and Figure 1 confirm the performance of the binary SCMWOA algorithm for feature selection problem.

B. BENCHMARK FUNCTIONS SCENARIO
This scenario tests the ability of the presented algorithm to get the best solution for the benchmark functions. Twentythree functions, divided into seven unimodal, six multimodal, and ten multimodal-based fixed-dimension benchmark functions, are employed in this sub-section. Figure 21 describe the unimodal functions parameters and range. The multimodal and multimodal-based fixed dimension functions description of the range and minimum values are shown in Figures 22  and 23. The SCMWOA in the second scenario is compared to original GWO [18], PSO [19], WOA [21], FEP [29], GSA [30], GA [25] algorithms, EGWO [31], hybrid of CSA and GWO (GWO-CSA) [31], and hybrid of SCA and PSO (bSCA-PSO) [28]. The mean and standard deviation (StDev) results of the suggested and compared algorithms over the benchmark functions (f 1 to f 23 ) are shown in Table 11. This table shows that the proposed SCMWOA algorithm achieved zero values in Mean and StDev in some cases and better results than the compared single and hybrid algorithms in other cases. Table 12 shows the ANOVA test for sample functions (f 1 , f 2 , f 3 , f 9 , f 11 , f 23 ). The T-test analysis test for all benchmark functions (f 1 to f 23 ) using the suggested algorithm against the compared algorithms is presented in Table 13.The histogram interpolation of a sample function (f 11 ) based on the SCMWOA algorithm and PSO, GWO, WOA, and GA algorithms is discussed in Table 14.
Box plot of the suggested and compared algorithms for benchmark function (f 1 to f 7 ), (f 8 to f 16 ), and (f 1 7 to f 23 ) are shown in Figures 2, 3, and 4, respectively. The histogram of the suggested and compared algorithms for benchmark function (f 1 , f 2 , f 3 , f 9 , f 11 , and f 23 ) are tested and shown in Figure 5. The Quantile-Quantile (QQ) plot of the suggested and compared algorithms for benchmark function (f 1 , f 2 , f 3 , f 9 , f 11 , and f 23 ) and the convergence curves based on the benchmark functions (f 1 , f 2 , f 3 , f 9 , f 11 , and f 23 ) are presented in Figures 6 and 7. The results of the proposed continuous SCMWOA algorithm in this scenario, compared to the state-of-the-art algorithms, confirm the performance of the algorithm for the benchmark functions.

C. SOLVING CONSTRAINED OPTIMIZATION PROBLEMS SCENARIO
This section is designed to validate the SCMWOA algorithm to solve two constrained optimization example of tension/compression spring and welded beam designs. The two engineering problems are described mathematically in equations 17-23. In addition, the SCMWOA algorithm results are compared with GWO [18], WOA [21], GSA [34], and PSO [19] algorithms result to get the minimum cost. Figure 8 shows the schematic diagram of tension/compression spring design (TCSD) [32]. TCSD is considered as a continuous constrained problem. The algorithm aims to minimize the volume of a coil spring under a constant tension/compression load. The TCSD has three design variables which are the number of spring's active coils, L, the diameter of the winding, d, and the diameter of the wire, w. The mathematical formulation of the TCSD can be described as follows: Minimize

1) Tension/Compression Spring design Problem
Subject to the following constraints where the three variables range are as follows: The box plot results of Tension/Compression Spring design based on different algorithms are shown in Figure 9. The histogram results of Tension/Compression Spring design based on different algorithms are discussed in Figure 10. Table 17 shows the comparison of one sample t-test analysis of the tension/compression spring design among other algorithms.
Tables 15 and 16 presents the best solution and the statistical results of proposed and compared algorithms for Tension/Compression Spring design Problem, respectively. The results of the proposed SCMWOA algorithm in this scenario compared to the state-of-the-art algorithms con-firm the performance of the algorithm for solving the Tension/Compression Spring design.

2) Welded Beam design problem
The next constrained problem is the welded beam design [33]. The schematic diagram of the welded beam design is shown in Figure 11. It is considered as an important benchmark to test different optimization methods. The main objective is to minimize the fabricating cost of the welded beam which comprised of the setup, welding labor, and material costs. The properties constraints are on the shear stress, bending stress, buckling load, end deflection, and the side constraint. Subject to the following constraints where   where the four variables range are as follows: The box plot results of the Welded Beam design problem based on different algorithms are shown in Figure 12. The histogram results of the Welded Beam design problem based VOLUME 4, 2016     on different algorithms are shown in Figure 13. Table 20 shows the comparison of the one sample t-test analysis of the welded beam design problem among other algorithms. Tables 18 and 19 presents the best solution and the sta- tistical results of proposed and compared algorithms for Welded Beam design problem, respectively. The results of the proposed SCMWOA algorithm in this scenario compared to the state-of-the-art algorithms confirm the performance of the algorithm for solving the Welded Beam design.

VI. CONCLUSION
This paper proposed an optimization algorithm called Sine Cosine hybrid with Modified Whale Optimization Algorithm (SCMWOA). The SCMWOA algorithm is tested using nineteen datasets, from the UCI Machine Learning Repository, with different number attributes, instances, and classes for feature selection. The SCMWOA algorithm is also tested for twenty-three benchmark functions. The functions include seven unimodal, six multimodal, and ten multi modal based fixed-dimension functions. The two tested engineering problems are the tension/compression spring design and the welded beam design. The results emphasize that the SCM-WOA algorithm outperforms several comparative optimization algorithms and provides high accuracy. Statistical analysis tests, including one-way analysis of variance (ANOVA) and Wilcoxon's rank-sum, confirm that the SCMWOA algorithm has better performance. The SCMWOA algorithm will be tested for more classical engineering design problems in future work since the algorithm perform well only in the two mentioned problems in this paper. Other benchmark functions, such as CEC 2015 and CEC 2017, will also be considered in future work.

APPENDIX
This appendix includes three tables of benchmark functions. Table 21 shows the description of the unimodal benchmark functions. Table 22 shows the description of the multimodal benchmark functions. Table 23 shows the description of the multimodal based fixed-dimension benchmark functions.