Image Mosaicing Based on Improved Optimal Seam-Cutting

To solve the problems of seams or ghosting caused by large exposure differences or moving objects in an image mosaic, this paper presents an image mosaicing method based on improved optimal seam-cutting. First, the feature points are extracted by the SIFT algorithm, and the false matching points are removed by the RANSAC algorithm. Next, a new energy function is constructed to perform optimal seam-cutting according to a dynamic programming algorithm that includes a colour and brightness data guarantee item in the YUV colour space and an eight-direction structural information guarantee item based on the Robinson operator. Finally, an improved weighted average method is proposed to fuse the images. The experimental results show that the method can make the transition of the image overlap area smoother and more natural, effectively eliminate the seam and ghosting and improving the image mosaic’s quality.


I. INTRODUCTION
The objective of image mosaicing is to align two or more images of the same scene with overlapping regions into a high-resolution wide-angle image. In recent years, image mosaic technology has been widely explored and applied. In the field of virtual reality, people can use the technology to obtain wide-angle images or 360-degree panoramic images that can be used to virtualize real-world scenes [1,2]. In medical image processing [3], the field of view of a microscope or an ultrasonic wave is small, and the doctor cannot grasp the full picture based on one image; data measurement of a large target image is needed to join incomplete images into a whole. Image mosaicing technology is a key link in enabling remote data measurement and remote consultation by stitching adjacent images together. In the field of remote sensing technology [4,5], two or more images of the same region can be compared using the image registration technique in image mosaic technology. Such technology can also be used to mosaic distorted ground images taken by remote sensing satellites into more accurate and complete images as the basis for further research. In the military field of night vision imaging technology [6], both night vision and infrared The associate editor coordinating the review of this manuscript and approving it for publication was Senthil Kumar.
imaging equipment are unable to acquire images with a wide field of view due to the limitations of camera equipment; by using the image mosaicing technology, an observer can observe everything around him or her. This technology also plays a very important role in infrared warning systems. The above observations show that the potential applications of image mosaic technology are very broad, and studying image mosaic technology in depth is very significant.
Mosaicing involves various steps of image processing: registration, reprojection, stitching, and blending [7]. Among these steps, image registration and blending are the two significant research areas that directly influence the image mosaicing performance. Registration refers to the establishment of geometric correspondence between a pair of images depicting the same scene. Registering a set of images requires estimating the geometric transformations that align the images with respect to a reference image within that set. The set may consist of two or more images of a single scene taken at different times, from different viewpoints, and/or by different sensors. The most general case of a transformation is the planar homography with 8 degrees of freedom [8]. The next step following the registration is reprojection, which refers to the alignment of images to a common coordinate system using the computed geometric transformations. The goal of the stitching step is to overlay the aligned images on a larger canvas by merging pixel values of the overlapping portions and retaining pixels where no overlap occurs. Image blending is the key technology used to resolve the exposure difference and eliminate the ghost image. In the process of image acquisition, due to the camera exposure time, environmental changes and other factors, the resulting images often differ in brightness and colour; if the images are simply stitched, there will be apparent stitching lines, colour inconsistencies, etc. If the images are of moving objects and some errors occur during registration, there will be problems, such as ghosting or the appearance of double shadows in the overlapped portions. Thus, a blending algorithm needs to be used during or after the stitching step to minimize the discontinuities and eliminate ghosting in the global appearance of the mosaic. Image blending algorithms include weighted average, multiresolution, Poisson, and optimal seam-cutting image blending algorithms. The weighted average algorithm performs well in real time, but it cannot eliminate the problems of ghosting, double shadows and exposure difference. The Poisson blending algorithm can effectively eliminate the stitching line and produces a natural transition, but the complexity of the algorithm is high, and running it is more timeconsuming. A multiresolution blending algorithm can also effectively eliminate the stitching line. This algorithm has a lower complexity and requires less time; however, it cannot blend images that do not strictly overlap. The optimal seamcutting blending algorithm can eliminate the ghosting and double shadows well, but it makes the object incomplete and produces significant image distortion that affects the quality of stitching if the seam line passes through a moving object or a region with a large registration error. In addition, if there is a large exposure difference, the stitched image will have an apparent stitching line.
The problems of the seam line passing through a moving object and a large registration error have been considered by several studies. In [9], in the context of the problem of ghosting caused by moving objects and inaccurate registration, the difference image is used to obtain weights, and it is extended and divided into regions, which improves the image quality after blending. As a result, the problems of ghosting and the visible stitching line are eliminated, but the stitching image displays a certain distortion. The position of feature points is weighted in [10], which makes the seam line pass through feature points first. However, the algorithm does not perform blending well if there is an error in image registration. In [11], the energy function was improved by morphological operation and colour saturation difference, but because of the poor choice of colour space, the problem of the seam line passing through a moving object could not be eliminated when there were many moving objects. To solve the problem of stitching with optimal seam-cutting, [12] optimized the stitching algorithm by using the image shape information to eliminate the stitching seam, but it cannot effectively solve the double shadow problem caused by the stitching line passing through the moving object. In [13], the overlapping regions are divided into three parts: left, middle and right. The difference between the grey value of the left part in the left image to be stitched, the right part of the pixel of the right image to be stitched, and the weighted average grey value of the pixel point in the two images is obtained. The grey value of the fused pixel is determined by comparing the magnitude of the difference with a threshold, and the grey value of the middle part of the fused pixel point is determined by its weighted average value. This method mitigates the problem of the stitching line to a certain extent. However, the resulting transition is unnatural.
To solve the problems of the seam line and ghosting caused by a large exposure difference or a moving object in image mosaic, this paper presents an image mosaic method based on improved optimal seam-cutting. Our contribution is as follow: 1) First,the optimal seam-cutting blending algorithm can better eliminate ghost and ghosting problems, but when the suture passes through a moving object or a region with large registration error, the object is incomplete, causing image distortion and affecting stitching quality. 2) Second, if there is a large exposure difference, the stitched image will have an apparent stitching line.
Therefore, this paper presents an image mosaic method based on improved optimal seam-cutting. First, the feature points are extracted by the SIFT (Scale-invariant feature transform) algorithm, and the pairings of incorrectly matched points are removed by the RANSAC (Random Sample Consensus) algorithm. Then, optimal seam-cutting is performed according to the dynamic programming idea, and a new energy function is constructed that includes the colour and brightness information guarantee item and the eight-direction structural information guarantee item. For the colour information guarantee item, we use the ''U'' and ''V'' channels representing chromaticity in YUV space to add E u (x, y) and E v (x, y) color intensities information. For the structural information guarantee item, the Robinson operator is used instead of the Sobel operator to obtain the gradient in eight directions; finally, an improved weight calculation method is proposed, and the image is fused by the improved weighted average method. The experimental results show that the method can make the transition in the image overlap area appear smoother and more natural, effectively eliminate the seam and ghosting, and improve the image mosaic quality.

II. CLASSIC OPTIMAL SEAM-CUTTING ALGORITHM
If image blending involves moving objects in the overlap area, ghosting may easily appear in the resulting stitched image. The optimal seam-cutting algorithm is an effective method of solving the problem. An ideal seam line should meet the following two requirements: 1) In terms of colour intensity, the difference in colour between the two images is the smallest, and 2) In terms of structural strength, the structural difference between the two images is the smallest. The best seam solution guidelines are as follows: where E brightness represents the difference in grey values of overlapping pixels. E geometry represents the difference in its structural values. E geometry is implemented by modifying the gradient calculation Sobel operator. The latter is computed as where S x and S y represent the templates of the 3×3 improved Sobel operator [15] in the x and y directions, respectively. When the improved Sobel operator is used for gradient calculation, the gradients in the x and y directions are calculated using templates According to this criterion, the idea of using dynamic programming to determine an optimal seam line is implemented as the following specific steps: Step 1. Initialization. The pixels in each column of the first row correspond to a stitch; the intensity value is initialized to the criterion value of each point, and the current point of the stitch is the column value of the stitch.
Step 2. Extension. A line for which the strength of the stitch has been calculated is expanded downward until the last line. The method of expansion is to add the current point of each stitch to the value of the 3-pixel criteria in the next line immediately adjacent to the point. For comparison, one of the three pixels of the next row corresponding to the minimum intensity value is taken as the extension direction of the seam; the intensity value of the seam is updated to be the minimum intensity value, and the current point of the seam is updated to be the minimum. The column of the next row in which the intensity value is located next to the pixel value.
Step 3. Selecting the best seam. The smallest seam among all the seams is selected as the best seam.  and the right image represents the first expansion after the seam has been initialized. The stitching of each pixel in a row emits 3 segments pointing to the next 3 pixels, and the solid segment indicates the minimum expansion direction of the stitch. As a result, the best line can be calculated when the last line has been calculated. The resulting seam achieves image stitching.

III. OUR IMPROVED METHOD A. IMPROVED OPTIMAL SEAM-CUTTING ALGORITHM
If image blending involves moving objects in the overlap area, ghosting may easily appear in the resulting stitched image. The optimal seam-cutting algorithm is an method of solving the problem. But if the image has obvious illumination difference or exists a plurality of moving objects, the stitched image still has suture or ghosting. In this paper, we presents an improved optimal seam-cutting.
The optimal seam formula is divided into two parts: brightness intensity information and structural strength information. Since the colour intensity in the energy function is the difference of the greyscale image, this paper adds two colour intensities using the "U" and "V" channels representing the chromaticity in the YUV space. As to the structural strength of the energy function, when the gradient calculation is performed, the Sobel calculation involves only the gradients in the x and y directions. Considering the correlation around the pixel points, the Robinson operator is used instead of the Sobel operator to obtain the gradient in eight directions. The energy function of the optimal seam is improved as follows: E(x, y) = E color (x, y) + ηE geometry (x, y) where E color (x, y) represents colour differences, and E geometry (x, y) represents structural differences. Our algorithm frame chart is shown in Figure 2.

B. IMPROVED COLOUR INFORMATION GUARANTEE ITEM
The optimal seam-cutting can effectively separate the moving parts, and the ideal optimal seam must minimize the colour difference. The original greyscale image is approximated as the colour difference intensity, and actually only considers the luminance information but is not considers to the colour information. In the YUV colour space, Y is the brightness (referred to as luminance or luma), which is the greyscale value, and ''U'' and ''V'' are the chroma (or chrominance), which is used to describe the colour and saturation of the image. ''U'' and ''V'' are the two components that constitute the colour. The significance of using the YUV colour space is that its luminance signal Y and chrominance signals U and V are separated. Accordingly, ''Y'' can be regarded as corresponding to the greyscale difference. It is natural to think that we can use the chromaticity ''U'' and ''V'' channels of the YUV space to add the corresponding chromaticity intensity. The effect is to increase the sensitivity of the stitch to the colour information in the image. Then, E color (x, y) includes three parts as follows: where E brightness (x, y) represents brightness differences, E u (x, y) and E v (x, y)) represent chromaticity differences, and α, β, γ and η represent the respective intensity weights and satisfy the condition α + β + γ + η = 1. Let a 1 and a 2 be the grey images of the overlapping area of I1 and I2 respectively; then, the greyscale difference image is calculated as To determine the difference in colour chromaticity, the image is first converted from the RGB space to U and V images in the YUV space, respectively. The conversion formula is as follows: Let U1 and U2 be the U images of the overlap area of I1 and I2, and V1 and V2 be V images of the overlap area of I1 and I2; then, The weights α, β and γ mentioned above should be selected based on a comprehensive analysis. Following an experiment, it was decided to set α = 0.3, β = 0.2, γ = 0.2, and η = 0.3, as such settings produced better results. We tested 8 cases with a pair of images. From the table 1, the parameters of case 1 are the best

C. IMPROVED STRUCTURAL INFORMATION GUARANTEE ITEM
The optimal seam can effectively separate the moving parts, and the ideal optimal seam must minimize the structural strength difference. When the gradient calculation is performed by the Sobel operator, only the gradients in the x and y directions are calculated. Considering the correlation around the pixel points, the Robinson edge detection operator is used instead of the Sobel operator to calculate the gradient in eight directions. To analyse the changes in the image in all directions, the following templates are used: Then, the solution formula is changed to Above, S 1 and S 2 . . . S 8 represent the respective templates of the 3 × 3 Robinson operator in eight directions.
D. IMPROVED WEIGHTED AVERAGE BLENDING [12] In this paper, the weighted average fusion algorithm [12] is used to achieve seamless splicing. In practical applications, the most commonly used stitching line smoothing method is the weighted smoothing method. The fusion is performed as shown in equation (10): Above, f is the fused image, f 1 and f 2 are two images to be spliced, and ω 1 and ω 2 are the weights of the pixels corresponding to the overlap region of f 1 and f 2 , respectively, that satisfy ω 1 + ω 2 = 1. The calculation of ω 1 , ω 2 follows equation (11): where x i is the abscissa of the current pixel, and x l and x r are the left and right boundaries of the overlap region, respectively, as shown in Figure 3(a). The weighted average of the  improved weighted average fusion algorithm is computed as shown in equation (12): Above, variable x i is the abscissa of the current pixel. Variables x min and x max are the left and right borders of the circumscribed rectangle of the best obtained stitch, as shown in Figure 3(b). The image and the circumscribed rectangular image with the boundaries of x l , x r and x min , x max , respectively, are shown in Figure 3. Using the recalculated weights ω 1 , ω 2 and the fusion formula (10) to smooth the stitched image can eliminate ghosting.

IV. EXPERIMENTAL RESULTS AND ANALYSIS
To assess the feasibility and performance of the proposed algorithm, we first experimented with three groups of images, including those of moving objects. Then, we used two sets of images with large exposure differences. The splicing results were compared with the traditional best seams and the results of other methods in [9]- [13]. To objectively illustrate the superiority of our algorithm, this paper uses the image mosaic quality evaluation indices [13] Cor and Errmg to evaluate the stitching effect. COR shows the similarity of the two images in the overlap region, Errmg represents the average geometric error, and the smaller the two values are, he better the stitching result is.

A. IMAGE STITCHING EXPERIMENT WITH MOVING OBJECTS
As shown in Figure 4, there are three sets of experimental images in which the objects in the images have clearly moved, and the splicing results are shown in Figure 5, Figure 6, and Figure 7. The best seams usually separate the moving objects well, but there are also cases of the moving objects being too close to the seam to cause poor fusion. The traditional best seams can separate the objects well, but the distance to the moving objects is very small. Ghosting can clearly be observed in the detail enlargement in Figure 5(b) and Figure 6(b). An existing algorithm of [9]- [13] produces has a good stitching for the first set of images, as shown in Figure 5(b) and there is no ghosting. However, in the second set of images that contain more moving objects, the stitches determined by the method do not successfully avoid such objects. Among all moving objects that may potentially result in ghosting, as shown in Figure 6(b), there is no apparent ghosting of the black images of pedestrians in the front, but images of the other two people exhibit clear ghosting; the results obtained by the algorithm proposed in this paper exhibit no ghosting in both places.
On the other hand, according to the data in Table 2, Table 3,the method is superior to the traditional method and several other methods in terms of parameter cor or parameter Errmg. Although the COR value in Figure 5 is slightly larger than that in the literature [10], the Errmg value is smaller than that in the literature [10], the parameters of the other two groups are smaller than those in the literature [10], and the stitching quality is superior to that in the literature [10].  These findings are is consistent with the results of our visual observations and also show that the method of this paper is superior to the existing methods [10]. The experimental results show that our method can effectively resolve the ghosting phenomenon caused by the close proximity of the best seam and moving objects and improve the quality of picture stitching.

B. IMAGE STITCHING EXPERIMENT WITH EXPOSURE DIFFERENCE
To further assess the effectiveness and performance of the proposed algorithm, in this section, the algorithm is tested by  selecting images with exposure difference. Figure 8 presents the original images in the experiment that clearly exhibit    brightness differences. The splicing results are shown in Figures 9 and 10. According to the data in Table 2, Table 3, the metrics of results produced by the algorithm of this paper are smaller than those of existing published methods regardless of whether the COR or Errmg metric is considered. Therefore, the experiment also shows that the proposed method not only can preserve the target image information well for images with moving targets but also eliminates the ghosting around the target, and it can also complete the fusion well for images with clear exposure differences. There is no apparent stitching line, and the stitching result of the algorithm is very good.
To further assess the effectiveness and performance of the proposed algorithm, we tested our mosaicing approach on the images of a dataset Czyste [2]. The stitching results are shown in Figures 11. From the data point of view, the result metric produced by the algorithm is smaller than the metric of the existing published method, and the mosaicing result of the algorithm is better.

V. CONCLUSIONS
In this paper, we propose an improved optimal seam-cutting method for image mosaicing. We construct a new energy function according to a dynamic programming algorithm, which includes the colour and brightness information guarantee item in the YUV colour space and the eight-direction structural information guarantee item based on the Robinson operator, and then also propose an improved weighted average method to fuse the images. The optimal stitching algorithm achieves a uniform transition in the image overlap area and eliminates ghosting. The proposed method is evaluated on a variety of image pairs with moving objects and exposure differences, and the results show that our method can preserve the target image information well, eliminate the ghosting around the target, and produce a merged image with no apparent gap. Our proposed method effectively solves the problems of large differences in exposure and moving objects. He is currently a Professor with the School of Technology, Beijing Forestry University, China. His research interests include forestry robot, computer vision, and image processing. VOLUME 8, 2020