A Novel Bundle Adjustment Approach based on Guess-aided and Angle quantization Multi-Objective Particle Swarm Optimization (GAMOPSO) for 3D reconstruction applications.

The 3D reconstruction process is very important in a variety of computer vision applications. Bundle adjustment has a significant impact on 3D reconstruction processes, namely in Simultaneously Localization and Mapping (SLAM) and Structure from Motion (SfM). Bundle adjustment which optimizes camera parameters and 3D points as a very important final stage suffers from memory and efficiency requirements in very large-scale reconstruction. Multi-objective optimization (MOO) is used in solving a variety of realistic engineering problems. Multi-Objective Particle Swarm Optimization (MOPSO) is regarded as one of the states of the art for meta-heuristic MOO. MOPSO has utilized the concept of crowding distance as a measure to differentiate between solutions in the search space and provide a high level of exploration. However, this method ignores the direction of the exploration which is not sufficient to effectively explore the search space. In addition, MOPSO starts the search from a fully randomly initialized swarm without taking any prior knowledge about the initial guess into account, which is considered impractical in applications where we can estimate initial values for solutions like bundle adjustment. In this paper, we introduced a novel hybrid MOPSO-based bundle adjustment algorithm that takes advantage of initial guess, angle quantization technique, and traditional optimization algorithms like RADAM to improve the mobility of MOPSO solutions; the results showed that our algorithm can help improve the accuracy and efficiency of bundle adjustment (BA).


I. INTRODUCTION
Fine-tuning is the procedure in which parameters of the system should be modified precisely to match specific observations. The solver of these problems usually starts from the initial values of the parameters and adjusts them to fit the model with given observations. A well-known finetuning technique called Bundle Adjustment is considered one of the most important procedures to improve the precision of the 3D reconstruction process. The bundle adjustment was solved as an NLSE problem using many iterative methods like Gauss-Newton, Levenberg-Marquardt, and others. These methods are sometimes vulnerable to fall in local minima especially when the initial solution has poor quality, so they become a bottleneck for computation and memory. [1] The Particle Swarm Optimization algorithm, which was inspired by the motion of bird flocks and schooling fish, was originally suggested by Eberhart and Kennedy in 1995. Since then, it has attracted the attention of many researchers around the world [2]. Due to its efficiency, PSO has been used in a variety of applications to date. A multi-objective version was proposed later [3] to adapt the algorithm to solve multiobjective optimization problems. Since then, many improvements were proposed to the MOPSO which enabled it to be used in different applications [4]. This research aims to improve the quality of the Bundle Adjustment process using an improved version of MOPSO. So, in this paper, we will present how to adapt the MOPSO algorithm to solve fine-tuning problems, and we will test the modified algorithm on standard multi-objective test functions as proof of concept, then we will use it to solve the bundle adjustment.
The key contributions of this study can be summarized as follows.
• proposing the concept of guess aided initialization to evolutionary algorithms to adopt them to fine-tuning problems. • Introducing a generalized version of the angle quantization concept and deploying it in MOPSO. • Proposing a novel approach for selecting the leader that can help to improve the exploitation abilities of the MOPSO based on hybridization with traditional gradient descent approaches. • Providing a new method to solve the bundle adjustment problem in a more efficient way than the traditional methods. The remainder of this paper is organized as follows. Section Ⅱ presents the background. While Section Ⅲ presents the proposed GAMOPSO algorithm followed by Section Ⅳ experiment sets for both standard MOO test functions and bundle adjustment, in Section V we will show the results and discussions, followed by Section Ⅵ which presents the summary and conclusions.

II. Background
Multi-Objective Optimization: Optimization problems that have at least two objective functions are called multiobjective optimization problems. These objectives sometimes conflict with each other. In other words, they are represented by different units of measurement and may have the same importance in the decision-making process. [1] Assuming that we want to minimize the objective functions, the problem of multi-objective optimization can be represented as follows: Min ( ) ≔ [ 1 ( ), 2 ( ) . . , ( )] (1) Subject to the following restrictions: → , = 1, 2, . . . , , = 1, 2, . . . , inequalities and equations of constraints imposed on the problem. A decision vector ̅ is called Pareto-optimal if it is not dominated by any other decision vector, the final result of the optimization problem is a set of Pareto-optimal vectors called Pareto-front. [2] Previous work: Conventional bundle adjustment methods are mainly based on the Levenberg-Marquardt algorithm or its improved versions, which mainly aim to reduce its computational complexity by manipulating the structure of matrices or using the reduced camera system and other methods. The Levenberg algorithm uses the sum of squares of the projection error as a cost function. So, for n cameras and m points Levenberg has O((m+n) 3 ) complexity and O((m+n)m.n) memory usage. Many methods have been introduced to reduce the complexity of BA, including the Schur complement method, which helps reduce the complexity to O(m 3 + mn) with memory usage O(m.n), and it can be reduced more depending on secondary sparse structures. [3] Despite the possibility of reducing complexity, iterative methods are often vulnerable to fall in local minima, especially if initial estimations are inaccurate, which negatively affecting the accuracy of the process of bundle adjustment process using it. Many researchers investigated the use of random search algorithms due to their ability to be immunized against local minima. The researchers in [4] used the particle swarm optimization algorithm (PSO) to find the rotation angles, they used the sum of the absolute projection error as an objective function. The algorithm helped reduce the number of control points needed to find the camera parameters, but the solutions were only for the camera orientation and the values were far from the ground truth values, which means a failure of the bundle adjustment. While in [5] the authors provided a bundle adjustment algorithm for aerial panoramic images based on the genetic algorithm and the objective function was the sum of the absolute value of the residual errors, they focused on estimating the orientation of the camera. In [6] the adaptive torque researchers used the PSO algorithm, in addition to the techniques of crossing over and mutations used from the Genetic Algorithm (GA) to increase the algorithm's exploration capabilities, and this algorithm is used to improve the initial estimations which later used to solve the bundle adjustment using the Gauss-Newton method. Researchers in [7] studied the performance of different optimization algorithms to solve the problem of bundle adjustment in reconstructing panoramic images. These algorithms include; Bat Algorithm BA, Gray Wolf Optimizer (GWO), Arithmetic Optimization Algorithm (AOA), Salp Swarm Algorithm (SSA), and Particles Swarm optimization algorithm (PSO). They concluded that the optimization algorithms gave better performance than the traditional Levenberg algorithm. However, this method is suitable for reconstructing panoramic images resulting from a purely rotational motion without translation. It is worth noticing that most of the previous works focused on the orientation of the camera or on special applications, and to overcome this shortage, we are proposing a novel bundle adjustment algorithm based on a modified version of MOPSO as illustrated in the next section.

III. Proposed algorithm:
The proposed algorithm is a modified version of MOPSO that can handle fine-tuning engineering problems. Algorithm 1 shows the basic steps of the Guess-aided, Angle quantization based Multi-Objective Particle Swarm Optimization (GAMOPSO) algorithm, where the algorithm starts the process of initializing the initial swarm based on the initial guesses and with a percentage of the swarm determined by the InitPercent parameter, as this percentage of the initial particles is created according to a Gaussian distribution in the adjacent area of the given initial guess. After that, the process of determining dominant solutions begins through the DetermineDomination method, where solution A is considered to dominate solution B if the values of all objective functions of solution A are better or equal to the corresponding values of objective functions of solution B, and there is at least one value of objective functions of A that is better than the corresponding value of the objective functions of solution B. To this extent, the initialization process for the initial swarm is finished, and the algorithm's iterative cycles begin as follows at each iteration do: 1-Create a copy of the current swarm and call it newSwarm 2-For each particle p in the new swarm apply the following steps: • Choose a leader for the particle • calculate the speed according to (4): • move the particle p according to (5): • Apply the mutation on the particle if it is within the mutationPercent portion, which gives a new particle p' • If p' is better than p, put p' instead of p in the new swarm • Structureless bundle adjustment: The first challenge developers face in any optimization problem is usually to formulate the problem relevantly, and, as we showed in the second section of this article, the representation of the bundle adjustment problem requires optimizing the camera variables and structure points, so the size of the optimized variable will be for n cameras and m points 6n+3m. When working on a large scale, we will have thousands of points and hundreds of camera positions at least, and therefore the use of the traditional formulation of the problem will have a very high computational cost, so an alternative representation could be used, which is found in [ 8 ] where they algebraically proved the possibility of eliminating structure points through the use of the three-view constraints shown in Figure (   : A two-view constraint that matches points from two scenes : A three-view constraint that matches points from three scenes.

3-Merge current swarm and newSwarm into
→ : The linear transition vector from frame k to frame l. R i : the rotation matrix of the camera i. K i : the matrix of intrinsic parameters of the camera i. P i : the positions of the pixels corresponding to the structure points in the camera i. This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and content may change prior to final publication. It can be seen that this representation greatly helps in reducing the size of the optimized variables, as we only optimize the camera variables instead of the camera variables and the structure points, and thus this representation will reduce the computational and memory requirements significantly. Further reduction was gained by sequentially processing the coming frames. In the first call of GAMOPSO, it processes the first three frames as in monocular bundle adjustment we can predict the position of the frames up to a specific scale, the first frame (x1) is fixed as the reference frame and frames (x2, x3) position and orientation were optimized relatively to x1 to minimize the three objectives functions (G1, G2, G3) as shown in figure (2-a).On On the other hand, when a new frame arrives, GAMOPSO will be called to optimize the position and orientation of the new frame (x4 for example) to minimize only two of the objective functions (G2, G3) as G1 was optimized in the previous call as shown in figure (2b).

Guess-aided initialization:
As the fine-tuning problems start with an initial solution that needs to be optimized, so to give GAMOPSO this ability a guess-aided initialization mechanism was proposed. Algorithm 2 shows the initialization process based on the initial guess, where a portion of the particles equivalent to the required ratio (InitPercent) is generated around the initial guess within specified standard deviation (STD) using a Gaussian distribution. After the required number of particles around the initial guess, the remaining particles are randomly generated along with the search space. The goal of initial guess is to help the algorithm to converge quickly as soon as possible; to understand the idea let us analyze the performance of the algorithm in all possible scenarios related to the quality of initial guess; actually, there are two scenarios: • For the first one we have an accurate guess, so part of the initial solution will be close to the optimal solution; thus the convergence of the algorithm will be faster than the random initialization case. • The second scenario is to have a bad guess, but as we have only a small portion of the solution initiated based on the guess and the greatest portion generated randomly throughout the space, in this case, the performance of GAMOPSO will be approximately equivalent to the performance of traditional MOPSO.  (2) and (3) illustrate the differences between the random and Guessaided initialization process.   This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and content may change prior to final publication. Angle quantization: The concept of angle quantization was proposed by [9] to improve the exploration process in the NSGA-II algorithm. Figure (4) illustrates the concept of angle quantization, as we can see that the algorithm has only selected the solutions with red color because they maintain equal distribution concerning angular sectors in addition to domination. So, the idea is to divide the objective space into equal angular sectors and calculate the number of solutions in each sector, then new solutions are chosen from the less explored sectors, which helps in a more balanced distribution of solutions over the whole searching space. The algorithm will begin by sorting the solutions into nondominated ranks. Next, each non-dominated rank is chosen one at a time to build the next generation, starting from the first rank, until reaching a rank at which the number of solutions exceeds the required number to complete the new swarm. In this stage, the algorithm will select the solutions that have a lower crowding angle range rank from the last rank. As a contribution of this research, the concept of angle quantization was adapted to the GAMOPSO algorithm and extended to be able to deal with more than two objective functions. To illustrate the idea, simply map the values of the objective functions on a plane specified by two objective functions each time, so that an independent crowding angle is generated between every two axes of the objective functions, as shown in Figure ( A probabilistic method was used to choose the plane to be explored each time we have to select from the same rank solutions, simply we chose one of the crowding angles based on uniform distribution with probability equals: = Using uniform distribution guarantees equal chances for exploration on each objective axis, thus improving exploration of the whole space. Selecting the hybrid leader: The traditional MOPSO selects the particle leader based on the roulette wheel, where a solution is chosen randomly from the repository, but this mechanism does not guarantee rapid convergence of the algorithm, especially in complex applications such as bundle adjustment, so we developed a hybrid mechanism that helps to boost convergence by integrating the RADAM algorithm [10], which is one of the latest and best gradient algorithms, so we choose one of the superior solutions in the previous cycles and apply gradient for a small number of iterations to improve its quality, and then use it as a leader for a proportion of swarm particles and thus deploy the effect of this solution, which in turn may help in discovering better solutions as the remaining particles move towards the leader. Basically, the third constraint -which was illustrated previously in equation (8)-contains elements from the first and second constrains; thus it could be a good absolute indicator of the solution quality and used as a selection criterion for the superior solution. After selecting the superior solution, RADAM is used to apply a gradient on the second constraint -because the first constraint will be constant as we illustrated in figure (2-b)using its derivative in (12) to compute the loss: Where: ′ : is the derivative of with respect to camera position and orientation. ′ → : is the derivative of the transformation from frame l to frame m with respect to camera position and orientation. ′ : is the derivative of with respect to camera position and orientation.
We used epsilon greedy method to balance between the random behavior of the algorithm (using a roulette wheel to choose the leader) and the deterministic behavior (using an improved solution as a leader) as follows: Figure 5. The concept of angle quantization based selection, we chose the solutions from the less explored angular sector. [9] This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and content may change prior to final publication.

IV. Experiment sets: • Standard multi-objective experiments description:
To investigate the effect of each suggested contribution, 3 variants of the GAMOPSO algorithm were implemented: • GAMOPSOD: which uses the crowding distance only to choose from the same ranked particles.
• GAMOPSOA: which uses angle quantization only to choose from the same-ranked particles.
• GAMOPSODA: which uses probability to select either crowding distance or angle quantization to choose from the same ranked particles. The three variants were compared with the MOPSO implementation suggested by [11]. A set of standard mathematical functions suggested for multi-objective optimization problems illustrated in Table  (1) was used to evaluate the performance of algorithms. These functions have been used in many previous studies in this field. Veldhuizen [12] used a set of these functions and two of them were selected (FON) [13], (KUR) in our case, in addition to (POL) [14] and (SCH). In 1999 Deb [15] proposed methodological approaches for developing test functions of multi-objective optimization algorithms, Zitzler and his colleagues [16] followed this methodology and proposed six optimization functions, five of them were used in this research ZDT1, ZDT2, ZDT3, ZDT4, ZDT6. Three standard MOO metrics were used to compare the performance:

1) Generational distance metric (GD):
This metric is given by the following formula: where |S| is the number of points in the Pareto set approximation and P is a discrete representation of the Pareto front. Generally, = 2. In this case, it is the Euclidean distance between the Pareto set approximation and the true Pareto. [17] 2) Hyper volume metric: Sometimes called an S-metric, the hypervolume is described as the volume of the space in the objective space dominated by the Pareto front approximation S and delimited from above by a reference point ∈ ℝ such that for all ∈ , ≺ . The hypervolume metric is given by: is the m-dimensional Lebesgue measure. [18]

3) Coverage of two sets metric (C):
Let A and B be two Pareto set approximations. The C-metric maps the ordered pair (A, B) to the interval [0; 1] and is defined as follows: ( , ) = |{ ∈ , there exists ∈ such that ⪯ }| | | The experiments were conducted 10 times to verify the performance of the algorithms, each time the seed of the random number generator was changed to ensure that the series of generated random numbers did not affect the results obtained by averaging the measurements of the ten experiments.

Bundle adjustment experiments:
To evaluate the performance of the proposed algorithm, bundle adjustment was applied to two artificial trajectories generated based on simulation of camera movement and projection of 3D points with noise to model the camera projection error [19]. The first trajectory consists of 200 points distributed over 20 overlapping frames, and the second consists of 200 points distributed over 50 overlapping frames. GAMOPSO was compared to the traditional Levenberg-Marquardt-based bundle adjustment algorithm [ 20 ] and the finite-deference approximation-based bundle adjustment [21]. The experiments were repeated five times using a different seed for the random number generator to statistically test the performance of the algorithm.

V. Results and Discussions
• Standard multi-objective results: 1) Generational distance: As can be seen in Figure (7), there is a significant difference in the generational distance between the GAMOPSO variants and the standard MOPSO, and since the less, the better, the proposed variants outperform the standard MOPSO in this metric. This means that the optimal solutions found by the GAMOPSO variants are closer to the ideal solutions. Also, sometimes GAMOPSOA outperforms GAMOPSOD, so in some cases angle quantization helped to improve the diversity of solutions more than the crowding distance, which helped the algorithm find more appropriate solutions. 2) Hyper-volume metric: Figure (8) shows the average hypervolume metric, which provides a qualitative measure of convergence and solution sparsity at the same time. The more the merrier, the results show that the proposed algorithms are superior to the MOPSO algorithm on the hypervolume metric; therefore, the quality of the solutions is better and its spread is more balanced, as it covers a larger area of the search space than the benchmark algorithm, which confirms the superiority of the proposed algorithm in all its variants over the benchmark algorithm in terms of hypervolume .  Figure (9) shows the average Coverage of the Two Sets metric, which calculates the percentage of solutions in a set A dominated by at least one of the solutions of another set B. The results show that the coverage ratio of the solutions found by the variants of the GAMOPSO algorithm ranged from 60% to 100%, compared to a low rate of opposite coverage that ranged between 0 and 5%, indicating that the quality of the solutions obtained from the proposed variants is better than the quality of the solutions obtained by the benchmark, which confirms the absolute superiority of the proposed algorithms over the benchmark MOPSO algorithm in this metric. Figure 9. Set coverage metric results: the coverage ratio of the solutions from the variants of the GAMOPSO algorithm ranged from 60% to 100%, compared to a low ratio of the opposite coverage ranging between 0 and 5%, which indicates that the quality of the solutions obtained from the proposed variants is better than the quality of solutions obtained by the benchmark. This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and

1) First trajectory results:
A camera trajectory was generated over 20 frames that detected 200 distinct points with large rotation steps between frames (6 degrees), as shown in Figure (10). A comparison was made between the proposed algorithm and the two benchmark algorithms (Levenberg-Marquardt and finitedeference approximation). The results showed an obvious superiority of the proposed algorithm in terms of accuracy and execution time. Figure (11) shows the box plot of the results of the first trajectory, where the red line expresses the mean error value for each algorithm during the experiment, while the blue rectangle represents the range of values within which the error ranged during the experiment. It is worth noting that the error values in the five experiments for each of the two benchmark algorithms are fixed as they are deterministic, while the proposed algorithm uses the random search technique, so there is a discrepancy in the error values according to the seed, despite that, the highest value of the error for GAMOPSO is less by 5% from the minimum error value of the benchmark algorithms.  (13) and (14) show the previous results presented in the form of columns, each column representing the result of the experiment using a specific seed on the X-axis.

2) Second trajectory results:
A camera trajectory was generated over 50 frames that detected 200 distinct points with smaller rotation steps between frames (3 degrees), as shown in Figure (15). A comparison was made between the proposed algorithm and the two benchmark algorithms (Levenberg-Marquardt and finite-deference approximation). The results showed an obvious superiority of the proposed algorithm in terms of accuracy and execution time. Figure (16) shows the box plot of the results of the second trajectory, where the red line expresses the mean error value for each algorithm during the experiment, while the blue rectangle represents the range of values within which the error ranged during the experiment. It is worth noting that the error values in the five experiments for each of the two benchmark algorithms are fixed as they are deterministic, while the proposed algorithm uses the random search technique, so there is a discrepancy in the error values according to the seed, despite that, the highest value of the error for GAMOPSO is less by 20% from the minimum error value of the benchmark algorithms. Figure (17) shows the execution time of the algorithms on the second trajectory, the proposed algorithm is faster and consumes approximately 56% of the time consumed by the Levenberg algorithm and approximately 58% of the time of finite deference. Figures (18) and (19) show the previous results presented in the form of columns, each column representing the result of the experiment using a specific seed on the X-axis.

VI. CONCLUSION
Bundle adjustment plays a vital role in 3D reconstruction processes. Therefore, improving the performance of bundle adjustment can be useful in a variety of applications, including Visual Simultaneously Localization and Mapping (VSLAM), Structure from Motion (SfM), Virtual Reality (VR), and others. So future research could examine the use of this algorithm in real 3D reconstruction applications like Visual Simultaneous Localization and Mapping and Structure from Motion. Or it might apply the same techniques proposed in this paper for GAMOPSO like guess-aided initialization, and hybridization with gradient descent algorithms to improve other evolutionary algorithms like (NSGAII, NSGAII, etc.).
In this paper, we proposed a novel bundle adjustment algorithm based on a developed version of MOPSO. The proposed algorithm used initial guess, angle quantization technique, and RADAM optimizer to improve the mobility and quality of the solutions, the results showed that our algorithm can help improve the accuracy and the efficiency of BA. Figure 10. The first trajectory of the camera 200 3D points, 20 frames, and 6 degree angular step Figure 11. The RMSE value for the entire trajectory as a boxplot: The highest value of the error for GAMOPSO is less than 5% than the minimum error value of the benchmark algorithms.
This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and content may change prior to final publication.   This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and content may change prior to final publication.     This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and content may change prior to final publication.