Learning-aided Evolution for Optimization

—Learning and optimization are the two essential abilities of human beings for problem solving. Similarly, computer scientists have made great efforts to design artiﬁcial neural network (ANN) and evolutionary computation (EC) to simulate the learning ability and the optimization ability for solving real-world problems, respectively. These have been two essential branches in artiﬁcial intelligence (AI) and computer science. However, in humans, learning and optimization are usually integrated together for problem solving. Therefore, how to efﬁciently integrate these two abilities together to develop powerful AI remains a signiﬁcant but challenging issue. Motivated by this, this article proposes a novel learning-aided evolutionary optimization (LEO) framework that plus learning and evolution for solving optimization problems. The LEO is integrated with the evolution knowledge learned by ANN from the evolution process of EC to promote optimization efﬁciency. The LEO framework is applied to both classical EC algorithms and some state-of-the-art EC algorithms including a champion algorithm, with benchmarking against the IEEE Congress on EC competition data. The experimental results show that the LEO can signiﬁcantly enhance the existing EC algorithms to better solve both single-objective and multi-/many-objective global optimization problems, suggesting that learning plus evolution is more intelligent for problem solving. Moreover, the experimental results have also validated the time efﬁciency of the LEO, where the additional time cost for using LEO is greatly deserved. Therefore, the promising LEO can lead to a new and more efﬁcient paradigm for EC algorithms to solve global optimization problems by plus learning and evolution.


I. INTRODUCTION
L EARNING and optimization are two basic abilities of human beings for problem solving [1].Similarly, artificial neural networks (ANNs) for learning and evolutionary computation (EC) algorithms for optimization are the two typical branches of artificial intelligence (AI), and both have obtained significant developments along with the development of computer science [2].In general, the ANN is a kind of connectionism AI that simulates the brain structure to assist machine in learning knowledge, while the EC is a kind of evolutionism AI that simulates the evolution phenomena and intelligent behaviors of humans or swarm animals for problem solving [2], [3], [4], both of which are essential tasks of AI.
Currently, the ANN-based learning branch has aroused great attention due to the success of deep learning (DL) in various real-world applications [5], [6], [7].More significantly, EC algorithms have also made the great pace in research and applications [8], [9], [10], [11], [12], [13].EC algorithms were born in the 1960s, when computer scientists designed EC algorithms, such as the genetic algorithm (GA) [12], [14], evolution strategy [15], and evolutionary programming [16], [17] for solving optimization problems.Since then, EC algorithms have attracted great attention and interest in the global optimization community.EC algorithms are promising because they do not require the strict mathematical characteristics of the problem and can find the global optimum or near-global optimum within an acceptable time.Generally, EC is a common framework that simulates the evolutionary mechanism of biology (e.g., GA [14] and differential evolution (DE) [18], [19], [20], [21]) and the swarm intelligence (SI) behaviors of animals/insects (e.g., particle swarm optimization (PSO) [22], [23], [24] and ant colony optimization (ACO) [25], [26], [27], [28]).The core idea of EC algorithms is "survival of the fittest."That is, new solutions are generated by simulating evolutionary phenomena, such as crossover and mutation.Then, the new solutions (i.e., offspring) and the old solutions (i.e., parents) compete to survive in the next generation.This is similar to the principle of natural selection in nature.By doing so, the solutions in the new generation are expected to be better than those in the old generations.Consequently, EC algorithms can gradually approach the global optimum generation by generation.The generic evolutionary framework of an EC algorithm is illustrated in the top of Fig. 1, where g is the generation index and will be increased by g = g + 1 after the selection until the algorithm termination criteria are met.However, solving the problem only by evolution may be inefficient.For example, in nature, evolution may need thousands of years to improve a species, while learning can help accelerate evolution dramatically.Concerning this, an insightful question is: Can we combine learning with evolution to solve complex problems more efficiently, just like the combination of the learning and optimization abilities of human beings?Moreover, as mentioned before, optimization using EC and learning using ANN are two main streams of AI, thus their combination is promising for designing powerful AI for achieving greater problem-solving ability.
In the history of AI, the two branches of EC and ANN mainly develop independently and sometimes interact with each other, resulting in the evolutionary neural network [29], [30], [31], evolutionary DL [32], evolutionary machine learning [33], neuroevolution [34], and machine learning-enhanced EC [35].However, few enhanced EC algorithms have employed ANNs to evolve solutions.As ANNs have already performed quite well in learning tasks, it is worth researching their ability to learn how to evolve well from the EC process to assist optimization tasks.In this way, the performance of EC algorithms can be surely improved by using ANNs.This can be a new way to design more intelligent systems for knowledge discovery and problem solving.Therefore, we propose a learning-aided evolutionary optimization (LEO) framework that can upgrade the performance of EC algorithms by plus using an ANN learning system.The LEO is a new EC paradigm that drives EC optimization using both the evolutionary fitness from problems and the learned evolution knowledge obtained by an ANN.The general idea of the LEO framework is given in Fig. 1.
As shown in Fig. 1, the learning system in LEO aims to learn knowledge from the EC algorithm process.Herein, the learned knowledge is the "evolution knowledge" that can guide the EC algorithm "how to evolve well" (e.g., to efficiently help the EC algorithm better approach the global optimum).To this aim, we collect evolutionary data during the EC process as an evolutionary pattern (EP) and use the ANN to learn from the successful EP (SEP) to obtain the evolution knowledge.Specifically, the EP of a solution can be represented by a pair of solutions that contains the original version of this solution in the current generation and the changed version of this solution in the next generation.In this sense, the SEP is defined as a successful pair of solutions whose changed version has better fitness than its original version.As shown in Fig. 1, in the learning system, if a pair of solutions (i.e., the old solution and the new solution) is successful, i.e., the new solution has a better fitness value than the old solution, then the relationship of this pair can be regarded as a kind of SEP that can help a solution evolve better.During the evolutionary process of the EC algorithm, many such successful pairs of solutions (i.e., SEPs) can be collected.Therefore, the learning system can be trained by using these collected SEPs, where the learned evolution knowledge from SEPs can be used to promote the evolutionary process.
In order to learn from these SEPs, in this article, the learning system in LEO is configured by using a very simple yet efficient ANN as an example.The training input of the ANN is the old solution of the successful pair while the output of the ANN is the corresponding new solution of the pair.After the training process with all the pairs, the ANN becomes a model that can help an input solution evolve better.That is, by learning a large number of SEPs, the ANN can be trained into a model that can receive a solution as input and then output a solution with better fitness.This is helpful to push the EC algorithms to better regions of the search space.Therefore, LEO can also be interpreted as "Learning + Evolution Optimization," which is an emerging and promising AI paradigm in the future EC era, as shown in Fig. 2. Fig. 2 also shows that the EC algorithms are different from and more efficient than the random search algorithms before the EC era due to the fitness-driven mechanism of evolution.Generally speaking, fitness information can be regarded as problem-related knowledge because it is designed by domain experts and is used for driving the evolution of the EC population.Nevertheless, the EC population also yields data and information during the evolution process, which can be learned and regarded as evolution-related knowledge to further enhance the EC algorithm.Therefore, in the future EC era, the next generational EC algorithms not only will be fitness driven (data and knowledge derived from the problem), but also will be learning aided (data and knowledge derived from the evolution), i.e., learning plus evolution, such as the LEO new paradigm proposed in this article.
In the experiments, the proposed LEO is evaluated on not only the single-objective optimization problems but also the multi-and many-objective optimization problems.To be specific, the single-objective optimization problems are actually the newest IEEE Congress on Evolutionary Computation (CEC) 2021 competition benchmark problems for single-objective real parameter numerical optimization [36].These benchmark problems consist of the most challenging complex optimization problems from the previous IEEE CEC competitions, e.g., IEEE CEC 2014, 2017, and 2018 competitions.Moreover, they are the most challenging and well-known benchmarks in the EC community.As for multi-/many-objective problems, the widely used benchmark suite, i.e., DTLZ [37], is adopted in this article with the number of objectives set as 5, 10, 15, and 20, respectively.Moreover, we apply the LEO framework not only to the classical algorithms for single-objective optimization (i.e., PSO and DE) and multiobjective optimization (NSGA-II [38] and MOEA/D [39]), but also to the state-of-the-art algorithms, e.g., the champion algorithm named NL-SHADE-RSP [40] on IEEE CEC 2021 competition.
The remaining contents are organized as follows.Section II gives a brief introduction of the EC, ANN, and related work.Section III details the LEO, while Section IV illustrates the experiments with results, comparisons, and analyses.Finally, the conclusion is provided in Section V.

A. Evolutionary Computation
EC algorithms can be divided into two categories, including evolutionary algorithm (EA) and SI [2].Herein, the following contents introduce two typical and widely used EC algorithms for optimization problems, i.e., the PSO [22] as a typical SI and the DE [18] as a typical EA.
1) Particle Swarm Optimization: PSO is a classical and efficient SI algorithm in the EC family [22].Modeling the behavior of bird flocks, PSO adopts a set of particles as a swarm to search for the global optimum.Each particle has a position vector to represent the candidate solution in the whole search space (i.e., the problem space) and uses a velocity vector to indicate the search direction.
First, the PSO initializes the positions and velocities of particles randomly within their corresponding upper bound and lower bound to represent their initial status.Then, the fitness values of all particles will be evaluated with the objective function of the problem.After the fitness evaluation, the personally best position (pBest) of each particle in their experience and the globally best position (gBest) of the whole swarm will be determined.Then, the PSO goes into the loop to conduct the evolutionary operators, such as velocity and position updates, new solution evaluations, and pBest and gBest updates, generation by generation.These processes are iteratively performed to evolve particles to find better solutions.Finally, when the algorithm meets the termination conditions, the algorithm will output the newest gBest as the final optimal solution.In particular, the velocity and position updates are performed according to (1) and (2), respectively where X i,j and V i,j are the jth dimension's position and velocity of the ith particle, respectively; ω represents the inertia weight; c 1 and c 2 are predefined coefficients; and r 1,j and r 2,j are two values randomly sampled in the range [0, 1].
2) Differential Evolution: DE is an easy-to-implement and efficient EA in the EC family.It treats candidate solutions as individuals, i.e., X i , and evolves individuals based on the individual difference.
After the random initialization of individuals in the search space, DE iteratively evolves individuals by three evolutionary operators until the stop criteria are met.These three operators include mutation, crossover, and selection, as introduced as follows.
Mutation: a mutation vector T i is computed for each individual X i with the position of other individuals in every generation."DE/rand/1" is a frequently used strategy for computing the mutation vector, which is listed as follows: where r 1 , r 2 , and r 3 represent the three randomly selected individual indexes different from each other and different from i, F is the mutation parameter.Crossover: A trial vector U i will be generated for each X i via crossover based on X i and T i .The widely used crossover operation is described as follows: where rand j is a value sampled from [0, 1] randomly, j rand is the random dimension index, which guarantees that U i adopts at least one dimension from T i , CR is the crossover rate.
Selection: In the selection, individuals with better fitness will be selected to enter into the next generation.Herein, the one-to-one selection in DE is considered, i.e., the better one between X i and U i will be selected.

B. Artificial Neural Network
As a learning system that simulates the human brain, ANN can map the input data to the targeted data after training the weights parameters of neurons based on the training data (i.e., input-output pairs).Therefore, given the successful evolution pairs of solution positions, the ANN can learn how to map poor solution positions to better solution positions through training.As this article adopts the multilayer feed-forward NN (a widely used ANN) [41] as the learning system in LEO, this part briefly introduces the multilayer feed-forward NN.
In general, a multilayer feed-forward NN contains several layers with corresponding neurons and connections.For example, Fig. 3 presents a multilayer feed-forward NN with N inputs and M outputs, where each connection (i.e., arrow) between the neurons in different layers means that the output of the neuron in the previous layer is the input of the neuron of the next layer.In addition, to provide the NN with nonlinear mapping ability, the activation function is used in the neuron of the hidden layer.In this article, we consider the widely used function, i.e., the sigmoid function, as the activation function in the neurons of hidden layers.Given the input as z, the sigmoid function can be written as follows: (5)

C. Related Work
To date, the research into using learning into EC has caused certain attention [35].According to the learning purposes, the existing works can be mainly categorized into learning the problem knowledge and learning the solution knowledge.
In the first category, learning the fitness as problem knowledge is a popular research topic.This is also known as datadriven EC because it uses evaluated individuals as data to learn the fitness for building a fitness surrogate, which is promising for solving expensive optimization problems [4].For example, Wang et al. [42] proposed a selective ensemble surrogate to learn the fitness knowledge efficiently.Li et al. [43] proposed a data-driven EC with a localized data generation method to learn the fitness knowledge from few-shot evaluated solution data.Also, to learn the fitness knowledge from few-shot evaluated data, Li et al. [44] proposed a multiple surrogate method based on data perturbation, which has obtained promising results.Moreover, Ji et al. [45] proposed a dual-layer cooperative surrogate model to learn the fitness knowledge of multimodal problems.Li and Zhang [46] proposed to use multiple local surrogates to learn the fitness and constraint knowledge from the constrained problems.However, datadriven EC mainly focuses on approximating the fitness and/or constraint evaluations via learning from the evaluated data.Differently, the proposed LEO mainly focuses on learning the SEP so as to help for better population reproduction and variation.Therefore, to better show their differences, the differences between the LEO and data-driven EC algorithms are briefly summarized in Table S.I of the supplemental material.
In the second category, the solution knowledge is learned mainly for the purposes of reusing past knowledge, transferring assisted knowledge, and predicting promising solutions.For example, Feng et al. [47] proposed to learn past knowledge from the solutions of the already solved vehicle routing problems and then reuse them to produce routing solutions for the current problem in a more efficient way.Zhou et al. [48] proposed to generate promising solutions via knowledge transfer from the promising solutions of other problems.Moreover, Feng et al. [49] proposed to predict the moving of optimal solutions by learning the dynamic change of the old optimal solutions in the old dynamic environments, so as to benefit the population reproduction in the current environment.Liu et al. [50] proposed a neural network-based information transfer method to transfer the solutions in the past environment to the promising solutions in the new environment.Also reusing knowledge of old solutions, Zhan et al. [51] proposed the adaptive distributed DE, where the promising individual positions in previous generations are maintained in the archive and then are reused for the individual evolution in the current generation.In addition, Ghosh et al. [52] proposed a difference vector reuse mechanism to reuse the successful differential vector learned from the solutions in the past generation to promote the evolution of the current population.Xia et al. [53] proposed a PSO with triple archives to store the promising particle positions as knowledge in different archives and reuse them to help better evolution.
The above methods have obtained potential results for improving the performance of EC, which suggests the significance of learning in evolution.Different from the above methods that focus on learning the problem knowledge and solution knowledge, the LEO proposed in this article aims to learn the evolution knowledge, i.e., how to evolve well, so as to promote population reproduction and optimization efficiency.

A. LEO Framework
The overall LEO framework is illustrated in Fig. 4, which contains two parts, i.e., the evolution part and the learning part.When compared with the traditional EC framework, the major novelty of LEO lies in that it uses not only the traditional evolutionary operator but also the learning-aided evolutionary operator to generate new promising individuals.In the learning part, the LEO trains the learning system with the successful solution pairs collected from the evolution part, so as to learn the SEPs.By learning from a large number of successful solution pairs, the ANN can learn enough SEPs to help achieve better individual evolution, which may be more straightforward and efficient than traditional evolutionary operators.
In the LEO framework, the learning system can be simple by using simple ANN (as studied in this article), and also can be complex if complex learning systems like deep networks are adopted (which can be studied in the future).Moreover, the learning-aided evolutionary operator is a generic operator and should be particularly designed based on the particular EC algorithm.The particular realization is described as follows.

B. Successful Solution Pairing
As data is significant for training the learning system, the quality of successful evolution pairs will greatly influence the performance of the learning system in learning-aided evolution.In fact, the definition of successful solution pair can vary, because that many individuals in the new generation can be regarded as the successful evolution of the individuals with worse fitness in the old generation.Without loss of generality, this article defines the successful solution pair as the SEP, which is described as follows.Given the ith individual in the generation g and g+1, denoted as X g,i and X g+1,i , respectively, (X g,i , X g+1,i ) is defined as a successful solution pair (i.e., a SEP) if the fitness of X g+1,i is better than X g,i .
Note that if in PSO, the X means the personal best position of the particle.That is, the successful solution pair (pBest g,i , pBest g+1,i ) is regarded as the SEP if pBest g+1,i is better than pBest g,i .Moreover, for multiobjective problems, the (X g,i , X g+1,i ) is defined as a successful solution pair if X g+1,i dominates X g,i .

C. Learning SEPs
The learning system in LEO aims to learn the SEP by using successful pairs of solutions.Therefore, in every generation, LEO collects all the successful pairs of solutions as training data and uses the collected data to update the ANN, i.e., trains the weights of the ANN.However, two issues should be considered in the data collection.The first is that the number of collected successful solution pairs will increase rapidly as the evolution goes on, and the second is that the SEPs learned in earlier generations may not be suitable for the solutions in later generations.With these two concerns, we use an external archive to store the successful solution pairs and update the archive periodically.By doing so, the number of available successful solution pairs can be adaptively controlled, and the out-of-date pairs can be naturally dropped.Specifically, after the data collection in every generation, LEO will check whether the number of collected successful solution pairs exceeds the maximum size of the archive, say arch_size.If exceeds, LEO only reserves the newest arch_size pairs and drops the old ones.Otherwise, the archive will not be changed.
With the data (i.e., the successful solution pairs) in the archive, the ANN can adaptively update its weights to learn SEPs better.Specifically, the ANN conducts one backpropagation (i.e., one epoch) in every generation with all the data in the archive to update its weights.The loss function of the ANN is the mean squared error between the network output and the target output, as can be written as follows: where (X i , X i ) is a successful solution pair stored in the arch, L(X i ) is the output of the learning system with the solution X i as the input.Note that other loss functions can be used if needed.
As the generation goes on, the ANN will conduct many epochs of training, and each training will use the newest successful solution pairs as the training data.

D. Learning-Aided Evolutionary Operator
The learning-aided evlutionary operator aims to use the learning system to drive the evolution of candidate solutions.By learning the SEPs, the learning system in LEO is able to know how to better optimize the position of an individual to enhance its fitness.That is, given an individual as an input, the learning system can output a new individual with better fitness.To fully utilize the learned evolution knowledge, we propose a learning-aided mutation (LM) operator and a learning-aided crossover (LC) operator.The LM operator can be written as follows: where X A , X B , and X C are three solutions as the inputs of LM, the L(X A ) is the output of the learning system with the solution X A as the input, and α is a parameter for the learningaided mutation.Moreover, the LC operator can be written as follows: where where X A and X B are the two solutions as the inputs of LC, j represents the dimension index, rand j is a random value within [0, 1], and β is a parameter in the learning-aided crossover.Based on the above, the LM and LC in PSO are implemented as follows: where pBest i is the personally best position of the ith particle, pBest r1 and pBest r2 are the personally best positions of two different random particles.As for DE, there are no personally best positions of individuals.Therefore, the individuals themselves in the current generation are used for implementing the LM and LC.That is, the LM and LC operators in DE can be rewritten as follows: where X i is the ith individual of the current generation, and X r1 and X r2 are the positions of two different random individuals in the current generation.Note that ( 7)-( 12) are examples to show how to design the learning-aided evolutionary operator, and without loss of generality, these operators can be substituted by other evolutionary operators with the output of the learning system [e.g., L(pBest i ) and L(X i )] to develop more efficient learning-aided evolutionary operator to enhance the LEO-based algorithms.

E. Combination of Traditional Evolutionary Operator and Learning-Aided Evolutionary Operator
As mentioned above, some individuals can be successfully evolved by the well-trained learning system via the learningaided evolutionary operator, which can be more efficient and straightforward than the traditional evolutionary operator.However, as the individuals and their successful evolution differ from generation to generation, the learning system needs to update itself adaptively with new SEPs in the latest generations.Otherwise, the learning system trained with out-of-date SEPs will become unsuitable for driving the evolution of individuals in current and future generations and stages.In order to collect more latest SEPs to update the learning system, the traditional evolutionary operator is still essential to explore new successful solution pairs.Therefore, LEO uses both the traditional evolutionary operator and the learning system to evolve the individuals instead of only using the learning system (i.e., only using the learning-aided evolutionary operator).With this concern, a parameter of learning probability lp is proposed for controlling the use of traditional evolutionary operator and learning-aided evolutionary operator in each generation.To be specific, for each generation, if a value randomly sampled within [0, 1] is smaller than lp, then the individuals will be evolved by the traditional evolutionary operator, so as to help explore the unknown landscapes to discover better successful solution pairs.Otherwise, the individuals will be evolved by the learning-aided evolutionary operator, so that the learned evolution knowledge can be well utilized.By doing so, the traditional and learning-aided evolutionary operator can work cooperatively to evolve individuals more efficiently.

F. Whole LEO-Based Algorithm
Herein, the whole LEO-based EC algorithm is presented in Algorithm 1.In the algorithm, there are five main procedures: initialization, individual evolution, selection, SEP collection, and learning system update.In the individual evolution, the learning system (i.e., ANN) will be used to evolve individuals via learning-aided evolutionary operators only when the index of the generation number is larger than one (i.e., g>1) and a randomly generated value is smaller than lp (i.e., r<lp), as shown in line 11 of Algorithm 1.The reason is that if g>1, then the ANN has been surely updated with the solution pairs in the archive, as shown in line 30 of Algorithm 1.In other words, the ANN will not be used to help the individual evolution until its first update, which can guarantee that the ANN has at least learned evolution knowledge from the successful evolution pairs once.After the individual evolution, the algorithm will carry out selection and SEP collection.In this procedure, the successful pairs of solutions are added in the archive arch as training data.The above four procedures (except the initialization) will be carried out iteratively until the algorithm meets the stop criteria.Lastly, the algorithm outputs the best-found solution as the result.

A. Experimental Settings
To evaluate LEO, experiments are conducted on not only single-objective optimization problems but also Train the ANN with all data in arch for one epoch; 31: End While 32: End multi-/many-objective optimization problems.Moreover, corresponding state-of-the-art single-objective and multi-/many-objective algorithms are used in the experiments.To be more specific, the experimental settings are given as follows.
1) Experimental Settings for Single-Objective Optimization: For experiments on single-objective optimization problems, the LEO-based algorithms will be tested on benchmark problems of the IEEE CEC 2021 Single-Objective Real Parameter Numerical Optimization Competition [36].This benchmark set, which is the latest and most well-known benchmark in the EC community, includes ten complex minimization problems that have been selected on purpose from the IEEE 2014, 2017, and 2018 CEC competitions [36], [40].As these problems span a wide variety of function characteristics, such as multimodal and nonseparable, they are suitable for investigating and evaluating the effectiveness and general performance of the proposed LEO framework from different aspects.More significantly, the well-known and typical properties of these problems will provide in-depth observations on how LEObased algorithms may behave in various situations.In addition, each problem can be set as a 10-or 20-D minimization problem, and the lower and upper bound of each dimension are -100 and 100, respectively, as recommended by the official of the CEC 2021 Competition [36].Therefore, in this article, each of the ten problems is set with 10 and 20 dimensions (i.e., 10D and 20D), and these 20 test problems are all adopted for the evaluation of LEO.In addition, on these benchmark problems, we apply LEO not only to PSO and DE, but also to the state-of-the-art algorithm NL-SHADE-RSP [40], which is a champion algorithm in IEEE CEC 2021 competition, to further investigate the effectiveness of LEO.Note that the population size of both PSO and DE is set as 100, while the NL-SHADE-RSP uses a dynamic population size due to its integrated population size reduction mechanism.To obtain fair comparisons, 2 × 10 5 and 1 × 10 6 are set as the maximum number of fitness evaluations for 10D and 20D problems, respectively, as suggested by the official of the CEC 2021 Competition [36].
2) Experimental Settings for Multi-/Many-Objective Optimization: For experiments on multi-/many-objective optimization problems, the LEO will be combined with the classical multiobjective algorithm NSGA-II [38], MOEA/D [39], and a state-of-the-art algorithm named RVEA [54].The LEO-based algorithms are tested on a widely used multiobjective benchmark suit, i.e., the DLTZ benchmark set [37].Note that as problem 8 and problem 9 in the DLTZ benchmark contain constraints, algorithms that are not designed for constrained problems, such as the MOEA/D, may generate infeasible solutions.Therefore, this article only uses the DTLZ1 to DTLZ7 to conduct the experiments.To further challenge the LEO-based algorithms, the number of objectives in each of the seven problems are set as 5, 10, 15, and 20.That is, 7 × 4 = 28 multi-/many-objective optimization problems will be used in the experiments.All the experiments are conducted with the maximum number of evaluations set as 1 × 10 4 for each problem.Furthermore, the settings of NSGA-II, MOEA/D, and RVEA are set according to the default settings in their original references, and the MOEA/D uses the PBI reference vectors.
3) Other Experimental Settings: As for the setting of LEObased algorithms, the arch_size is set as 100, while α and β in the learning-aided evolutionary operator are set as 0.5 and 0.9, respectively.The learning system ANN is set with three layers, which include one input layer, one hidden layer, and one output layer.As both the input and output of the ANN are candidate solutions, the dimension of the input and output layer will be D, where D is the problem dimension.Besides, the number of neurons in the hidden layer is set as 3 × D. Note that the input and output values of the ANN should be in the range of [0, 1].Therefore, each dimension value of solutions will be transformed from the original search space to [0, 1] to be the input of ANN and the output values of the ANN will be transformed from [0, 1] back to the corresponding search space to be the generated individual.For the training and implementation of ANN, as mentioned earlier in Section III-C, the LEO-based algorithms will randomly initialize the ANN at the beginning and then perform one epoch of backpropagation in every generation with all the data in the archive.To reduce the statistical errors, each algorithm will be conducted 30 times on each single or multi-/manyobjective optimization problem independently, so as to get the statistical results for the comparison.Besides, this article uses Wilcoxon's rank sum test with a significance level of α = 0.05 to determine whether the algorithms with LEO are significantly better than, statistically similar to, and significantly worse than the algorithms of the original versions on each problem in terms of the optimization accuracy.In addition, the Friedman test with Holm post-hoc analysis with a significant level of 0.05 is also used to compare the algorithm performance on multiple problems.Based on this, the compared algorithm performs significantly differently from the proposed algorithm if the corresponding adjusted p-value is larger than 0.05, otherwise, the two algorithms have similar performance on multiple problems.Note that, all the LEObased EC algorithms are named as L-EC where EC is the original name of the algorithm.To quantify the performance improvements, the effect size is computed based on Cohen's d statistic [55] and reported in this article.Similar to existing work [56], the effect size is regarded as: small if 0.2≤d<0.5;medium if 0.5≤d<0.8,or large if 0.8< d, which are denoted with symbols "s", "m", or "l", respectively.

B. Comparisons on Single-Objective Optimization Problems
Table I provides the comparison results among LEO-based algorithms and their original versions on single-objective optimization problems.In Table I, the LEO is very effective in improving the optimization efficiency of EC algorithms.As can be seen in Table I, L-PSO, L-DE, and L-NL-SHADE-RSP can significantly outperform PSO, DE, and NL-SHADE-RSP on 19, 10, and 7 problems, and only have worse results on 1, 2, and 0 problems, respectively.Moreover, although NL-SHADE-RSP is the champion algorithm in the CEC competitions with these test problems, the L-NL-SHADE-RSP can still obtain better optimization results and update the optimization result records obtained by NL-SHADE-RSP on seven problems, i.e., F04(10D), F07(10D), F10(10D), F04(20D), F06(20D), F07(20D), and F20(10D).This suggests the great effectiveness and practical potential of LEO.Overall, the LEO is promising for improving EC algorithms on single-objective optimization problems.

C. Comparisons on Multi-/Many-Objective Optimization Problems
Table II gives the comparison results among LEO-based algorithms and their original versions on multiobjective optimization problems.The results indicate that the LEO is also very effective in improving the optimization efficiency of EC algorithms on multi-/many-objective problems.As shown in Table II, L-NSGA-II, L-MOEA/D, and L-RVEA can obtain significantly better IGD results than NSGA-II, MOEA/D, and RVEA on 14, 9, and 6 problems, and have worse results only on 8, 8, and 5 problems, respectively.Moreover, Table II shows that the NSGA-II has more improvement than the MOEA/D and RVEA when integrated with LEO.This may be due to that when the number of objectives increases, the number of the nondominated solution increases dramatically and the NSGA-II (i.e., the nondominated selection-based algorithm) is difficult to conduct environmental selection to select high-quality solutions into the next generation for better evolution.However, the LEO can directly evolve solutions toward the promising region, and therefore can improve the NSGA-II significantly.Overall, the results have shown the effectiveness of LEO on multi-/many-objective optimization problems.

D. Comparisons on High-Dimensional Optimization Problems
To examine the proposed LEO on more higherdimensional problems, this part compares L-PSO, L-DE, and L-NL-SHADE-RSP with PSO, DE, and NL-SHADE-RSP, respectively, on optimization problems with 100 dimensions.The number of fitness evaluations for different algorithms is set the same as 1 × 10 6 .Table III gives the comparison results between LEO-based algorithms and their original versions.As can be seen in Table III, L-PSO, L-DE, and L-NL-SHADE-RSP can significantly outperform PSO, DE, and NL-SHADE-RSP on 10, 8, and 6 problems with 100 dimensions, respectively, but have worse results on none of the problems.Moreover, according to the p-values, L-PSO, L-DE, and L-NL-SHADE-RSP have significantly better overall performance than PSO, DE, and NL-SHADE-RSP on the ten problems.Therefore, the experimental results further verify the superiority of the LEO on higher-dimensional problems.

E. Time Efficiency Analysis of the LEO
The above results have experimentally verified the effectiveness of LEO in improving the optimization results.Therefore, this part further investigates the time efficiency of the LEO for solving optimization problems.To study the time efficiency of LEO, this article proposes a novel indicator metric named error reduction versus time increase rate (i.e., ETR), which can measure the benefit of additional time cost when using LEO (i.e., can be treated as a measurement of cost performance).Mathematically, given two algorithms A and B, where A requires more computational time cost than B but has better performance than B, the ETR of A over B is defined as follows: where ERR(A, B) is the optimization error reduction rate of A over B and the TIR(A, B) is the time cost increase rate of A over B. For a minimization optimization problem, the ERR(A, B) can be computed as follows: where E A and E B are the optimization error of A and B, respectively.If ERR(A, B) is 0, then A has no error reduction over B, i.e., no accuracy improvement when compared with B.Moreover, the TIR(A, B) can be calculated as follows: where T B and T A represent the computational time cost of A and B, respectively.Therefore, ETR(A, B) can measure how many percentages of optimization error reduction A can obtain over B for each increased percentage of time cost.Based on the ETR, this part studies the time efficiency of LEO by analyzing the ETR of L-PSO over PSO in different situations.To be specific, this part analyzes the ETR(L-PSO, PSO) on F01 to F10 at 10D with five computational time cost settings in fitness evaluation (i.e., the time cost of each fitness evaluation in the five scenarios requires 0.1, 0.5, 1, 1.5, and 2 ms, respectively).The five settings of time cost are implemented by using a time-delay operation in the fitness evaluation.For brevity and simplicity, the following contents simply refer ETR as ETR(L-PSO, PSO).The ETR results are plotted in Fig. 5, and the running time of L-PSO and PSO are also given in Table S.III in the supplementary material.
From Fig. 5, we can have two important observations.First, the ETR increases nearly linearly as the time cost of fitness evaluation increases on most problems.This suggests that the more expensive the fitness evaluation cost is, the more efficient the LEO method will be (one percentage increase in time cost can bring in more percentages of error reduction).In fact, the computational time cost of one fitness evaluation in various real-world applications can be very expensive, e.g., several hours to several days [4], which are much more expensive than the highest settings in this experiment (i.e., 2 ms).Second, when the time of each evaluation is larger than 0.5 ms, the ETR is very remarkable, and when the time cost of each evaluation is 2 ms, the ETR is about 14 on most problems, i.e., every increased percentage of time cost when using LEO can result in about 14% of optimization error reduction.These results have shown the great potential and advantage of LEO.In addition, the running time given in Table S.III in the supplementary material shows that the L-PSO just requires a small percentage of additional running time when compared with PSO when the time cost of fitness evaluation is larger than 1 ms.Based on the above, it can be concluded that the additional time cost for using LEO is considerably economical and deserved, and the LEO is very efficient in turning the computational cost into the performance improvement of the algorithm.
In addition, to provide a fair comparison from the aspect of the same running time, this part further compares the LEO-PSO, LEO-DE, and LEO-NL-SHADE-RSP with their original version under the same running time.To do this, the running time of all algorithms are set the same as 240 s, i.e., 4 min.Similar to the setting for Fig. 5, the time cost of each fitness evaluation is set as 1 ms controlled by a time-delay operation.The comparison results are given in Table S.II of the supplementary material.As shown in Table S.II in the supplementary material, when using the same running time, L-PSO, L-DE, and L-NL-SHADE-RSP can significantly outperform PSO, DE, and NL-SHADE-RSP on 9, 5, and 3 problems, and only have worse results on 1, 1, and 0 problems, respectively.Moreover, according to the adjusted p-value, the L-PSO and L-DE are significantly better than PSO and DE over the multiple problems, respectively.Although the L-NL-SHADE-RSP and NL-SHADE-RSP have similar results over multiple problems based on the adjusted p-value, the L-NL-SHADE-RSP can still outperform NL-SHADE-RSP on F04, F07, and F10.Based on the above, the LEO-based algorithms are still efficient when compared with the baseline algorithms with the same running time.

F. Effectiveness of Learning-Aided Evolutionary Operator
This article proposes to use the learning-aided evolutionary operator with ANN to generate promising individuals, rather than directly using the output solutions of ANN to make up the new population.Therefore, this part studies whether using the output solutions of ANN directly can work.To do this, a new L-PSO variant named ANN-PSO is developed, where during the learning-aided process (refer to line 12 of Algorithm 1), the learning-aided evolutionary operator is not used while the output solution of ANN is used directly as the new individual.That is, (7) and ( 8) are not used.
The comparison results between L-PSO and ANN-PSO are given in Table IV.As can be seen, according to the adjusted pvalue, the L-PSO is significantly different from the ANN-PSO on the multiple test problems.Moreover, the L-PSO greatly outperforms ANN-PSO on 8 of the 10 problems.In addition, the L-PSO obtains the best results on all ten problems.These results show the great superiority of L-PSO over ANN-PSO.This may be due to that although an ANN is used to learn how to generate promising individuals in the ANN-PSO, the ANN is misled by the local optima found in the early evolutionary stage and then produce new individuals around local optima, which results in the poor performance of the ANN-PSO.Differently, the learning-aided evolutionary operator used in L-PSO can generate new individuals based on the individuals produced by ANN and the individuals in current populations, which has more diversity and can enhance the optimization results.Overall, the learning-aided evolutionary operator is effective for enhancing the algorithm results.

G. Influence of Learning Probability
To study the influence of lp in LEO-based algorithms, this part compares the L-PSO with its variants using different lp values.That is, the L-PSO(lp = 0.5) is compared with L-PSO(lp = 0.1), L-PSO(lp = 0.3), L-PSO(lp = 0.7), and L-PSO(lp = 0.9).The comparison results are shown in Table V.In general, according to the adjusted p-value, the L-PSO(lp = 0.5) has no significant difference from the rest four variants over the ten problems.This shows that  the proposed LEO is not sensitive to the learning probability.Moreover, L-PSO(lp = 0.5) can generate significantly better results than L-PSO(lp = 0.1), L-PSO(lp = 0.3), and L-PSO(lp = 0.9) on 4, 5, and 3 problems, and produce worse results only on 2, 0, and 3 problems, respectively.Moreover, when compared with L-PSO(lp = 0.7), L-PSO(lp = 0.5) has a similar performance in general, e.g., has similar results on eight problems.The above results may be due to that both too small lp (e.g., lp = 0.1) and too large lp (e.g., lp = 0.9) will deteriorate the algorithm performance and an appropriate lp (e.g., lp = 0.5 or 0.7) can make a good balance between the traditional evolutionary process and learning-aided evolutionary process to obtain better results.Moreover, the experimental results also suggest that lp = 0.5 can be a good setting, and therefore lp = 0.5 is recommended in this article.

I. Influence of Learning Parameters
This part studies the influence of two learning parameters, i.e., α and β.To begin with, to study the influence of α in LEObased algorithms, this part compares the L-PSO with its variants with different α values.That is, the L-PSO(α = 0.5) is compared with L-PSO(α = 0.1), L-PSO(α = 0.3), L-PSO(α = 0.7), and L-PSO(α = 0.9).The comparison results are shown in Table VII.In general, according to the adjusted p-value, the L-PSO(α = 0.5) has no significant difference from the other four variants over the ten problems.Moreover, the L-PSO(α = 0.5) performs similarly to L-PSO(α = 0.1), L-PSO(α = 0.3), L-PSO(α = 0.7), and L-PSO(α = 0.9) on 8, 9, 9, and 10 problems, respectively.This shows that the LEO is not sensitive to the setting of α.Moreover, as L-PSO(α = 0.5) can generate the best results on six problems (as marked in boldface) among the five algorithms, which is the most, the α = 0.5 is used in this article.
Moreover, this part also studies the influence of β by comparing the L-PSO with its variants with different β values.That is, the L-PSO(β = 0.9) is compared with L-PSO(β = 0.1), L-PSO(β = 0.3), L-PSO(β = 0.5), and L-PSO(β = 0.7).The comparison results are shown in Table VIII.As can be seen, the L-PSO(β = 0.9) significantly outperforms L-PSO(β = 0.1) and L-PSO(β = 0.3) on 3 and 5 problems, and performs similarly to L-PSO(β = 0.5) and L-PSO(β = 0.7) on 9 and 8 problems.This suggests that larger β can yield better results.This may be due to that a larger β encourages the individuals to learn more dimensions from the individual produced by ANN, which enhance the optimization efficiency.In addition, according to the adjusted p-value, the L-PSO(β = 0.9) has no significant difference with L-PSO(β = 0.1), L-PSO(β = 0.5), and L-PSO(β = 0.7) over the multiple problems, and only have significant difference with L-PSO(β = 0.3).Therefore, the L-PSO is not that sensitive to the β value.

J. Influence of ANN Hyperparameter
This part studies the influence of the ANN hyperparameter.The L-PSO is compared with its variants using different numbers of hidden neurons.That is, the L-PSO(N h = 3 × D) is compared with L-PSO(N h = 1 × D), L-PSO(N h = 2 × D), L-PSO(N h = 4 × D), and L-PSO(N h = 5 × D), where N h denotes the number of hidden neurons and D is the problem dimension.The comparison results are shown in Table S.IV of the supplementary material.As can be seen in Table S.IV in the supplementary material, the L-PSO(N h = 3 × D) performs similarly to L-PSO(N h = 1 × D), L-PSO(N h = 2 × D), L-PSO(N h = 4 × D), and L-PSO(N h = 5 × D) on 10, 9, 10, and 10 problems, respectively.Moreover, according to the adjusted p-value, the L-PSO(N h = 3 × D) performs similarly to the rest four variants.That is, the results produced by L-PSO variants with different numbers of hidden neurons are very similar.This may be due to the fact that the ANN is a powerful learning system and can still learn the evolution knowledge efficiently when using a slightly different hyperparameter.Overall, the LEO is robust and is not sensitive to the network hyperparameter, and the original setting (i.e., N h = 3 × D) is recommended.

V. CONCLUSION
In this article, an innovative LEO framework for EC algorithms has been proposed.In the LEO, the ANN is adopted as the learning system to learn the SEPs collected from the evolutionary process, and then the learned evolution knowledge is applied to promote the solution evolution.As a result, the candidate solution can evolve directly toward a more promising region, which is more straightforward and efficient than traditional evolutionary operators.
To evaluate the LEO framework, the performance of some state-of-the-art and champion algorithms are compared with those of their LEO-based variants on both single-objective and multi-/many-objective optimization problems.In these experiments, the LEO-based algorithms achieve better performance than those without LEO.The experimental results show that solutions can not only avoid the dilemma of being unable to extract enough information to enhance evolution but also achieve faster convergence speed via the LEO framework, especially when encountered with high-dimensional problems.Moreover, a novel ETR indicator has been proposed to measure the time efficiency of the LEO, and the simulation experimental results have shown that the additional time cost for using LEO is greatly economical and deserved, especially when the fitness evaluation is very expensive.Therefore, although the concept and core idea of the proposed LEO framework is simple and easy to understand, it is very effective and efficient, which leads to a novel and more efficient paradigm for EC algorithms in solving global optimization problems.Moreover, the ETR indicator is a general metric that can be promoted in the EC community for measuring the cost performance of different EC algorithms.
Nevertheless, there may still have some room for enhancing the proposed LEO framework and LEO-based algorithms.First, the difficulties of complex problems may pose new challenges to the LEO-based algorithms, e.g., multiobjective problems with complex Pareto fronts [57] and a large number of objectives [58], which should be well-studied.Second, if the fitness evaluation of the problem is expensive to access, there will be only few-shot solutions that can be evaluated with the real fitness evaluation.In such case, how to efficiently use the limited training data should be well-studied.
Therefore, in the future, we will not only enhance the LEO-based algorithms to well address the above issues, but also hope to extend the idea of knowledge learning to more research aspects of EC and real-world application problems, such as large-scale problems [59], [60], [61], multimodal problems [62], [63], and multitask problems [64], [65].Moreover, the automatic optimization of the network architecture and parameter of the LEO learning system for different scenarios will be worthy of study [66].Additionally, the implementation of the LEO with powerful computation techniques, such as distributed computing [67], [68], is also a potential research direction for future work.

Fig. 3 .
Fig. 3. Example of multilayer feed-forward neural network with N inputs and M outputs.

Fig. 5 .
Fig. 5. ETR of L-PSO over PSO on different functions at 10D under different time costs in each fitness evaluation, where the larger ETR value means the better efficiency.

TABLE I COMPARISONS
BETWEEN LEO-BASED ALGORITHMS AND THE ORIGINAL ALGORITHMS ON SINGLE-OBJECTIVE OPTIMIZATION PROBLEMS

TABLE II COMPARISONS
OF IGD RESULTS BETWEEN LEO-BASED ALGORITHMS AND THE ORIGINAL ALGORITHMS ON MULTIOBJECTIVE OPTIMIZATION PROBLEMS TABLE III COMPARISONS BETWEEN LEO-BASED ALGORITHMS AND THE ORIGINAL ALGORITHMS ON PROBLEMS WITH 100 DIMENSIONS

TABLE V COMPARISONS
AMONG L-PSO VARIANTS WITH DIFFERENT LEARNING PROBABILITIES

TABLE VI COMPARISONS
AMONG L-PSO VARIANTS WITH DIFFERENT ARCHIVE SIZES FOR COLLECTING TRAINING DATA