Quasi-Dense Matching Algorithm for Close-Range Image Combined With Feature Line Constraint

Point-based sparse or dense matching can typically obtain satisfactory 3D point clouds of general contour features, but the deformation problem at the edges of artificial objects is prominent. Thus, to ensure the regularity of straight line edges, a quasi-dense matching algorithm for close-range images combined with feature line constraint is proposed in this study. The method utilizes reliable matched points to construct the initial Delaunay triangulation and then optimizes the triangulation using the matched feature line. On this basis, iterative quasi-dense matching based on triangulation constraint is implemented. First, the center of the inscribed circle of each triangle is used as the seed point for matching based on triangulation and epipolar line constraints. Then the successfully matched seed points are used for region growing while each growing point is matched. The triangulations are updated, and the aforementioned process is repeated until no new matched points are generated. Finally, tracking matching based on the quasi-dense matched points is performed on image sequence and 3D coordinates of matched points are calculated. Two sets of stereo image pairs acquired using smartphones and four sets of image sequence provided by public datasets are selected for quasi-dense matching experiments. The comparison of results of constraint matching of the two triangulations before and after optimization as well as the matching results obtained via VisualSFM system demonstrated that the 3D point cloud obtained via quasi-dense matching with feature line constraint has more regular edge points and better integrity, thereby confirming the effectiveness of the proposed algorithm.


I. INTRODUCTION
Image matching aims to establish the correspondence between corresponding points on images acquired from different viewpoints of the same object scene and is the first crucial step in acquiring three-dimensional (3D) information on the basis of images and then realizing 3D reconstruction [1]. This method is widely applied in computer vision and other fields, such as 3D vision, robotics navigation, image stitching, gesture recognition, and video tracking.
According to the selection of matching primitives and densities of matching results, image matching can be The associate editor coordinating the review of this manuscript and approving it for publication was Li Zhang . divided into three categories: feature-based sparse, feature propagation-based quasi-dense, and pixel-by-pixel dense matching. Sparse matching algorithms commonly use features detected by MSER [2], Hessian affine [3], and SIFT [4], [5], [6], [7] as matching primitives given their distinguishable and stable characteristics, and matching can obtain hundreds of corresponding points. Although these algorithms can be used to calculate internal and external parameters of the camera and recover general outline features of the object, insufficient details fail to display the comprehensive information of the target object. Dense matching generally refers to pixelby-pixel matching and is generally used for short baseline images acquired by binocular stereo vision or epipolar images obtained by correcting wide baseline images. Almost every pixel can obtain its corresponding pixel, except for reasons such as occlusion, where correspondence fails to establish. However, dense matching is typically applied to epipolar images, and all pixels are regarded equally while ignoring the advantage of image features. On the basis of these two aspects, quasi-dense matching has been proposed. Quasidense matching starts from the sparse seed point matching and grows the matching in the seed point neighborhood to obtain a uniformly distributed and sufficiently abundant number of matched points. Quasi-dense matching takes into account the advantages of feature and dense matching, in which seed point selection and seed point-based matching propagation are the two main aspects.
Matched points obtained by the sparse feature matching algorithm are generally used as seed points for quasi-dense matching in the seed point selection. Commonly used feature points include Harris corner points [8], Hessian affine detection points, SIFT detection points, ASIFT detection points [9], KLT corner points [10], and maximally stable extremal regions on difference of Gaussian space (MSER-DoG) region centroids [11].
Seed point-based matching propagation presents the following main aspects: On the one hand, matching propagation is performed within the seed point neighborhood. Lhuillier et al. [12] and [13] sorted the initial sparse matches using ZNCC scores during feature point matching and adopted the best-first propagation strategy to find new matches in the seed neighborhood by combining local constraints. This process considers the neighborhood of seed points on the two images as satisfying a simple translational transformation. Therefore, the algorithm is unsuitable for wide baseline images. Accordingly, Kannala et al. [14] proposed a quasi-dense matching algorithm for wide baseline images on the basis of a local affine model. This method matches feature points extracted via the Hessian affine operator to obtain initial seed points, normalizes local regions on the two images using the local geometric transformation matrix calculated from seed points during the matching propagation process, and calculates ZNCC scores between correlation windows. This method solves the imaging disparities and geometric deformations on different images caused by viewpoint changes to some extent and achieves satisfactory quasi-dense matching results for wide baseline images. However, multiple solutions are typically matched for images with single and repetitive textures in the matching propagation process. Megyesi et al. [15] put forward a matching algorithm that considers local affine transformation and parametric propagation for wide baseline images with rich texture. The algorithm uses the matching result of feature points detected using the KLT algorithm as the initial seed point and maintains the growing points within the local smoothing region of the current seed points by combining parallax and affine transformation constraints. The optimal parallax and affine transformation parameters are selected for matching propagation until the boundary of the smoothed region is reached and growing matching in the current region is stopped.
However, the input of this algorithm is epipolar images and the algorithm is unsuitable for texture-poor images. Xu et al. [16] employed Megyesi's algorithm for quasi-dense matching of fisheye images and achieved satisfactory results. Chen et al. [11] ranked seed points according to the adaptive support weight (ASW) score proposed by Yoon et al. [17], adopted the best-first growing principle, and used adaptive support weighting to calculate the matching score during the matching propagation process to remove matching ambiguity and solve the problem of matching multiple solutions caused by repetition.
On the other hand, geometries are constructed using initial matched points for image segmentation, and seed points to be matched and matching propagation are determined on the basis of the segmented region. Commonly used geometries are Voronoi diagrams [18], [19], [20] and Delaunay triangulation [21], [22], [23]. Li et al. [24] utilized the Voronoi diagram for constraint propagation to achieve dense matching. Similarly, Delaunay triangulation is often used as propagation carriers and widely used for dense matching [25], [26], [27], [28]. Existing research has focused on using corresponding triangles on two images to constrain the matching search range [29], [30], which can be combined further with the parallax constraint on vertices of the triangle [31], [32]; geometric relationship between the point to be matched and triangle vertices to constrain the matching process or eliminate false matches [33]; or equal proportion points of corresponding edges to obtain corresponding points [34]. Wu et al. [35] proposed a combining point and line matching method based on self-adaptive triangulation constraint, wherein feature lines are added to the triangulation for constraint matching to improve the matching performance of images in poor textural regions. However, this study still essentially a sparse matching algorithm since only using feature points and lines for matching.
In summary, research on quasi-dense matching has demonstrated some progress. In terms of the above-mentioned seed point-based matching propagation, the former only takes into account the initial matched point quality, while the latter considers the graphical constraint. However, the edges of the Voronoi diagram or Delaunay triangulation are different from the feature lines on the image, which makes it difficult to ensure the parallax continuity in the segmentation region and affects the significance of the graphical constraint to some extent. Notably, the 3D point cloud obtained on the basis of point dense matching can properly show the contours of objects although problems with bending and deformation still exist at the edges of objects, especially for buildings. Therefore, a quasi-dense matching algorithm for close-range image combined with feature line constraint is proposed in this study. Compared with the triangulation constructed only using matched points, this algorithm establishes Delaunay triangulation by combining existing matched points and lines and uses feature lines as sides of the triangle to constrain the matching growing and propagation and optimize the effect of the point cloud at edges of objects. The algorithm specifically includes four main aspects: initial delaunay triangulation construction and optimization, seed point matching, seed point growing and growing point matching, and image sequence tracking matching and 3D point acquisition.
The remainder of this paper is organized as follows. The principle of quasi-dense matching based on the triangulation constraint and 3D point cloud acquisition based on image sequence, including triangulation optimization with feature lines, seed point matching and growing, and growing point matching, are introduced in Section II. A comprehensive analysis of the proposed method and comparison of the results of the proposed method with those of the VisualSFM system are discussed in Section III. Finally, the conclusions and future studies are presented in Section IV.

II. QUASI-DENSE MATCHING BASED ON DELAUNAY TRIANGULATION CONSTRAINT AND 3D POINT RECONSTRUCTION
The flowchart of the proposed algorithm is shown in Figure 1. The two stages of the proposed algorithm are quasi-dense matching and 3D point reconstruction. The first stage is the main focus of this study. An image sequence is regarded as the input, and a stereo image pair consisting of the first two images of the image sequence is selected as the initial pair for dense matching. ASIFT and RANSAC algorithms are used to match the two images to obtain the initial reliable matched points. Meanwhile, the line matching algorithm [36] is used to match the two images to obtain corresponding lines. The initial corresponding triangulations on both images are constructed using the matched points, which are then optimized via segmental insertion of matched lines into the triangulations. On this basis, a quasi-dense matching algorithm with iterative triangulation constraint is proposed. Determine whether the radius of the inscribed circle for each triangle in the triangulation is greater than the given threshold and then use the center of the inscribed circle, which presents a radius greater than the given threshold, as the initial matching primitive for quasi-dense matching. This successfully matched circle center is regarded as the seed point, region growing within the circle is carried out according to the grayscale similarity, and each growing point is matched. The triangulation on each image is updated by adding new matched points obtained from the matching process to the current triangulation. These steps are repeated until the radius of the inscribed circle of all triangles in the triangulation is less than the given threshold or new matched points stop appearing and the iteration is halted.
Tracking matching is performed on an image sequence to ensure the stability of the camera's attitude parameters and obtain accurate 3D point clouds. The quasi-dense matched points from the first image pair are used as the matching primitive for subsequent image sequence matching, and Kanade-Lucas-Tomasi algorithm is utilized for tracking and matching until all images are loaded to obtain tie points between adjacent images. The camera coordinate system of the first image is considered the world coordinate system. External orientation elements of the camera, including camera rotation R i and translation t i matrices, are calculated on the basis of matched tie points obtained from tracking matching, and 3D coordinates of matched points are calculated using triangulation method in SFM [37]. As sequential images are added, the above process is repeated until all images have been tracked and the initial 3D point clouds are obtained. Bundle adjustment optimization algorithm is applied to improve the resulting point clouds and avoid the effects of error accumulation during the iterative process.

A. INITIAL DELAUNAY TRIANGULATION CONSTRUCTION WITH FEATURE LINE OPTIMIZATION
Delaunay triangulation is the carrier of the matching process that presents the following roles in the overall algorithm: First, the center of the inscribed circle of the triangle is used as the initial seed point. Second, the triangle is adopted to constrain region growing and growing point matching. Finally, triangulation is iteratively updated by inserting new matched points into the triangulation to ensure that the size and area of the triangle are progressively reduced as well as constrain the density of matches. Therefore, triangulation plays a crucial role in the algorithm. Triangulation is established using both feature point and line primitives to improve its accuracy at edges of the object and use it to constrain the following quasi-dense matching. The initial Delaunay triangulation is constructed using reliable matched points and then further optimized by inserting matched feature lines into the triangulation in segments. To construct corresponding triangulations on both images, triangulation on the reference image is first constructed as a benchmark using the following principle. Figure 2(a) shows a local part of the triangulation constructed using reliable matched points, and the yellow line is the matched feature line to be added to the triangulation. This line is used as an example. First, determine which triangles in the triangulation the line goes through (triangles 3, 234, 966, and 991 in this case), as shown in Figure 2(b). Second, judge the relationship between the line and each triangle through which it goes and record endpoints of the line segment and its intersections with triangles that divide the line into several segments, as indicated by black dots in Figure 2

B. SEED POINT MATCHING
The algorithm in this paper regards the center of the inscribed circle of triangle as the seed point for quasi-dense matching and the range of the inscribed circle as the growing region of the seed point. Epipolar, inscribed circle, similarity, and geometric constraints are used in the seed point matching process in turn. The specific implementation process is presented as follows: Step 1. Select any triangle on the reference image and determine whether the radius of the inscribed circle of the triangle is larger than the given threshold value T R . If so, then the center T riη of its inscribed circle is used as the seed point for matching.
Step 2. Determine the set of matching candidates for the point T riη to be matched as = p j , j = 1, 2, . . . , n . The fundamental matrix F is obtained using the eight-point method with initial corresponding points, and the epipolar line of the point T riη is calculated using l = F·T riη . The epipolar line intersects its inscribed circle at points a and b within the corresponding triangle on the search image. Pixels through which the line segment with endpoints a and b on the epipolar line passes are the initial set of matching candidate points 0 for the point T riη , as illustrated in Figure 3.
Step 3. Calculate the value of the normalized cross correlation (NCC) between point T riη and each candidate point in the set 0 separately, with a correlation window size of 7 × 7. Retain the candidate point with an NCC value larger than the given threshold T ρ for the set of matching candidate points = p j , j = 1, 2, . . . , n , where n is the number of candidate points in the set . Step 4. If n is greater than 0, then the set is not empty. Calculate the absolute value of the Euclidean distance differ-ence between each candidate point in the set and interse-ction points a and b and then denote it If min ({D 1 , D 2 , . . . , D n }) < T D is satis-fied, then the candidate point with the minimum distance difference is selected as the corresponding point of T riη and recorded as T riη . Note that T riη , T riη is a pair of correspond-ing points.

C. SEED POINT REGION GROWING
The successfully matched point T riη (red pixel in Figure 4) is used as a seed or reference point for region growing, and the gray difference constraint is utilized to perform crossshaped region growing within an inscribed circle with the center T riη . As shown in Figure 4, the seed point serves as the center, and region growing is carried out pixel by pixel in each of the four directions: horizontally leftward, horizontally rightward, vertically upward, and vertically downward. The absolute value of the gray difference between the pixel to be grown and the seed point is calculated. If the value is less than the given threshold T g , then growing continues in that direction until the absolute value of the gray difference is greater than T g or the growing range reaches the inscribed circle boundary to form a cross-shaped region, as shown in blue in Figure 4. Each pixel on the long arm of the cross-shaped region is treated as a postgrowing seed point that grows in the direction perpendicular to the long arm on both sides until the absolute value of the gray difference between the pixel to be grown and the postgrowing seed point is greater than T g or until it reaches the boundary of the circle. Finally, the growing region with T riη as the seed point is produced, and new growing

D. GROWING POINT MATCHING
Each growing point is selected for matching in turn in the set obtained by the cross-shaped region growing. The epipolar, inscribed circle, and similarity constraints are used in turn in this process to determine matching candidates, as mentioned in Section B, assuming that any point T i riη to be matched is tested with these three constraints to obtain the set = q j , j = 1, 2, . . . , n of matching candidates, where n is the number of candidate points. If the set is not empty, then the geometric constraint is further used to determine the final corresponding point. Different from the geometric constraint in seed point matching, this process considers a more complex relative position relationship on the basis of the matched seed point involving the orientation and distance constraints from the matching point to the matched seed point on both images.
Rotation and scale transformation parameters between both images are first calculated using the corresponding triangula-tions to avoid the influence of rotation and scale change betw-een two images in the matching process. On the basis of corresponding triangles in the corresponding triangulations, the length ratio and angle difference of each pair of corresponding sides are calculated and the value with the maximum frequency in each type of result is selected as the scale transformation rate δ and rotation angle θ between the two images, respectively.
. . , m , and mode (·) represents the function that finds the value with the maximum frequency. On this basis, the geometric constraint is implemented as follows: Step 1: Connect the matching point on the reference image to T i riη to obtain the line segment L. Connect each candidate point in the set of matching candidates on the search image to T riη to obtain the line segment set = L j , j = 1, 2, . . . , n . Calculate the length ratio and the rotation angle of the line segment L and each line segment in the set and denote them as sets = δ j , j = 1, 2, . . . , n and set = θ j , j = 1, 2, . . . , n , respectively.

E. ITERATIVE PROCESSING FOR QUASI-DENSE MATCHING
The three steps of seed point matching, seed point region growing, and growing point matching are carried out for each triangle in the triangulation. The triangulation is updated with new matched points and the above three steps are repeated for the newly generated triangles until no new matched points are generated or the radius of the inscribed circle of all triangles is less than the given threshold T R after all triangles are processed. The iterative matching is then stopped.

III. EXPERIMENT AND ANALYSIS A. CONSTRAINT MATCHING EXPERIMENT WITH DELAUNAY TRIANGULATION BEFORE AND AFTER FEATURE LINE OPTIMIZATION
Stereo image pairs of building walls with line features were mainly obtained to verify the effectiveness of the proposed method. The two sets of experiment images were obtained using a VIVO X7 smartphone camera, which was calibrated using the Zhang Zhengyou method [38], as shown in Figure 5. The size of images was 4160 × 3120. The number of reliable corresponding points obtained by initial matching was 2182 and 7394. Delaunay triangulations were cons-tructed using initial corresponding points, as shown in Figure 6. The following parameters in the proposed quasi-dense matching method based on experiments are set: T D = 1, T g = 15, T δ = 3, and T θ = 3.
An overlay of triangulations constructed with initial corresponding points and matched lines on the reference image is shown in Figure 7. Yellow points are intersection points of matched lines and triangulations or end points of matched lines. Each matched line was inserted into the triangulation in segments according to intersections, with each subsegment inserted as an edge of a new triangle, resulting in the triangulation shown in Figure 7. The final corresponding    triangulations optimized with feature lines on two sets of images are obtained (Figure 8).
The corresponding triangulations constructed from matched points and the corresponding triangulations after matched feature lines optimization were used as the basis for matching respectively, and the proposed iterative quasi-dense matching algorithm was used to match the stereo images. The same parameters were used in the quasi-dense matching process for different stereo image pairs. In order to compare and analyze the results of quasi-dense matching more intuitively, the 3D coordinates of the corresponding points obtained using the above two strategies were solved separately.
The process was solved using the principle of projection matrix. First, fundamental and essential matrices were estimated using the initial reliable matched points. Second, the essential matrix was decomposed to obtain the camera pose. Third, camera projection matrices were acquired. Finally, the linear least square method was adopted to solve the 3D coordinates of matching points and obtain the 3D point clouds.  The 3D point cloud results for the two stereo image pairs were observed from the main, bottom, and side views (Figures 9 and 10). Topological relationships between different facades of the target can be easily observed although differences exist in details of point clouds obtained by different strategies. As can be clearly observed from Figure 9(a), there are obvious irregularities in all corner lines. This is due to the fact that the construction triangulation based only on sparse feature points produces the region crossing phenomenon at the edge, i.e., a triangle contains different facades of the target object with texture discontinuity phenomenon. Although this limitation is improved by a similarity constraint of gray difference between neighboring pixels in the process of seed point growing and propagation, achieving an effective constraint for cases where different facades contain similar gray information is difficult.
Triangulation after matched feature line optimization is used in this work as the basis for quasi-dense matching to solve this problem. The matching results of two sets of stereo image pairs are shown in Figures 9(b) and 10(b). The comparison of Figures 9(a) and 9(b) demonstrated that the 3D point clouds obtained from the triangulation optimized by matched lines are better at the edges and the distortion and deformation at the edges and corners of the building are improved. The comparison of Figures 10(a) and 10(b) showed that dense point clouds obtained from both triangulations can better represent edge lines of building because the initial ASIFT matching obtains abundant dense matched points at intersection lines of different facades, as shown in Figure 6(b). The difference between the two types of triangulation constraint matching approaches is mainly in the coverage of point clouds. Compared with the coverage of corresponding points, matched feature lines increase the coverage of quasi-dense matching and then extend the coverage of the resulting point clouds.
For the initial triangulation and triangulation optimized by matched feature lines, the number of initial matched points and the number of final matched points were shown in Table 1. Compared with the number of matched points, the addition of matched lines increases the number of initial corresponding points by 190 and 1736. At the same time, the triangulation optimized by matched lines for constraint matching increases the number of matched points by 258831 and 356848, respectively, compared with the triangulation constructed by matched points. This analysis demonstrated that the two advantages in using feature lines for triangulation construction and constraint matching are (1) the effective improvement of edge details of the target object while reducing the phenomenon of distortion and deformation at the edge and (2) extension of coverage of matched point clouds when the distribution of matched points is clustered.

B. QUASI-DENSE MATCHING RESULTS OF IMAGE SEQUENCES USING DIFFERENT METHODS
Four sets of representative image sequences from the Lund University website were selected for quasi-dense matching experiments. Image sizes of the first three groups were 1296 × 1936, and the size of the fourth group was 1600 × 1200. Figure 11 displays only nine images in the sequence although 12, 14, 18, and 11 images are available for Door, Fountain, Statue, and Sculpture, respectively.
The proposed algorithm was applied to perform quasidense matching for the four sets of images using the two types of triangulations in Section III.A as the basis for matching.  The radius and NNC thresholds were set to T R = 5 and T ρ = 0.75, respectively. In addition, quasi-dense matching results from the open-source system VisualSFM were utilized for comparison analysis. The resulting point clouds for different algorithms are shown in Figures 12 to 15.
The results showed that VisualSFM can rapidly reconstruct 3D point clouds of image sequences and appropriately reflect the general contour of the object despite the presence of holes or points missing, as demonstrated by yellow boxes in Figures 12 to 15. The large point cloud coverage obtained by the proposed algorithm and satisfactory point cloud completeness can improve the hole phenomenon because the region growing of seed point can lead local neighborhood points to participate in matching. Hence, point clouds are rich in detail, especially for regions with abundant texture. For example, the lampstand in the Statue and the background wall in the Sculpture are better displayed than the results of VisualSFM system.
Compared with the triangulation constructed only from matched points, the triangulation constructed by combining the matched point-line for constraint quasi-dense matching presents the following advantages. The addition of feature lines can better constrain the matching effect at edges of objects to ensure that growing matching is performed in disparity continuity regions, reduces false matches, and regularizes the edge point cloud. The 3D point clouds from different views of the two datasets Door and Fountain (Figures 12 and 13) demonstrated that image edge information is enriched and the appearance of jaggedness at the edges is optimized. Point clouds of Door are better and more accurate after adding feature lines. Point clouds of Fountain at the connection between the edge and the lampstand are clearer and more accurate, thereby providing a better display of the object's shape. As for Statue and Sculpture, there  track matching point clouds both increase significantly after optimization with matched feature lines. The former Statue was mainly reflected in the surrounding area of the building, whereas the latter Sculpture was mainly reflected in the local detail area, where local point clouds are clustered in blocks and richer in detail information.
The number of matched points, the number of iteration times, and running time were counted for further analysis. The statistical results are shown in Figures 16 to 18. Figure 16 shows that the number of initial corresponding points for the four sets of images obtained through the ASIFT algorithm are 10644, 4343, 12285, and 3855, and those participating in the construction of triangulations increase by 11954, 789, 3746, and 4698, respectively, after feature line optimization. The process can increase the number of initial corresponding points, optimize triangles at the edge, and finally raise the number of quasi-dense matched point clouds. Fountain and Statue images demonstrate abundant local features and a small gray difference that significantly increased the number of matched points. The number of initial dense points of Fountain and Statue images increased 2.01 and 1.08 times while that of final matched points obtained from multiview image tracking matching increased from 337819 to 1204240 and 370040 to 910319, respectively, after optimization of feature lines. Figure 16 presents that the number of point clouds after multiview tracking matching decreases compared with the number of initial dense points. The reason is that when feature points are tracked and matched between image sequence, the KLT optical flow tracking algorithm will eliminate some unstable points, resulting in the improvement of matching reliability. Figures 17 and 18 exhibit that iteration and running times also increase significantly with the addition of feature lines. This phenomenon occurs because of the following reasons.
(1) The addition of feature lines expands the initial feature region of some image pairs, that is, the matching coverage increases. (2) The addition of feature lines increases the number of triangles in the triangulation. Although the VisualSFM system presents many advantages in the rapid reconstruction of 3D point clouds of image sequence, detailed information and local holes are lacking. The proposed quasi-dense matching algorithm can enrich the detail information through iterative matching with triangulation constraint and seed point region growing. Moreover, the addition of feature lines enhances the constraints of point    matching at the edge of the building, regularity of 3D point clouds at the edge and corner of the building, and plane features of point clouds.

IV. CONCLUSION
A quasi-dense matching algorithm that combines feature line constraints for close-range images is presented in this study. The proposed algorithm adds the matched feature line to the triangulation constructed using matched points to constrain the quasi-dense matching, which improves the appearance of point clouds at edges of artificial objects, such as buildings. The results revealed that, compared with the triangulation constructed by only matched points, the triangulation constructed by combining point-line features in this study is used to constrain matching, which improves the matching effect at edges of objects, and achieves enhanced repre-sentation of edge features with 3D point clouds. Notably, triangulation is used as the carrier of the proposed matching process and considered the original data for match propa-gation and constraint matching. Hence, ensuring the correct-ness of initial matched points and lines is necessary. More-over, during the process of inserting the matched line into the triangulation in segments, the correspondence between divi-sion segments of corresponding lines in both images must be established using the epipolar line. Therefore, a follow-up investigation will focus on accurately establishing the corres-pondence in the case where the line is nearly parallel to the epipolar line.