Automatic Deformation Extraction Method of Buildings in Mining Areas Based on TLS Point Clouds

Presently, the deformation extraction of buildings in mining areas using terrestrial laser scanner (TLS) point clouds is performed manually. Automatic deformation extraction holds great significance for building deformation monitoring in mining areas. Therefore, this study proposes an automatic extraction method for building deformation in mining areas using TLS point clouds. The corner points of doors and windows on the wall are considered as key points and the wall deformation in the mining area is extracted with minimal manual intervention. First, the input data were preprocessed, including 2D boundary point cloud acquisition and denoising (using a distance slope filter). Next, the key points were extracted via three steps: boundary line splitting, seed key point clustering, and key point judgment. Finally, the 3D coordinates of the key points and the relationship between the key points of the two phases were established to calculate the deformation value. The results confirmed negligible difference between the deformation value extracted using this method and the real value. Most of the errors were between -5 and 5 mm, and only a few exceeded ±5 mm; however, no error exceeded ±9 mm. The deformation value obtained using this method was almost identical to that obtained using the manual method, and the absolute error between them was below 8 mm. The performance verification of this method showed that the proposed distance slope filter removed the noise points more effectively compared to that of the traditional denoising filters, i.e., statistical and radius filters, establishing its suitability for the complex measurement environment of mining areas. During the automatic extraction of key point coordinates, the root mean square error (RMSE) values were below 2.0 mm; RMSE values during manual extraction were below 7.1 mm. The proposed method demonstrated greater stability than that of the manual extraction method.


I. INTRODUCTION
After mining of an underground coal seam, the rock stratum around the mining space loses its support and gradually begins to move. With the continuous expansion of the mining working face, this movement process gradually spreads to the surface, causing serious damage to the surfaces of buildings [1]- [3]. Therefore, extracting the deformation of buildings in mining areas to provide a basis for mining damage assessment has always been trending in deformation monitoring [4], [5].
In recent years, with the development of surveying and mapping technology, terrestrial laser scanners (TLS), known for their high precision and high efficiency, have been widely used in deformation monitoring, such as bridge monitoring and tunnel monitoring [6]- [9]. Reference [10] proposed a method to extract the deformation features of the highway bridge head-based TLS point cloud for bridge monitoring. This method includes an automatic data acquisition system, an efficient and accurate denoising method for TLS point clouds and can effectively eliminate the random error of key point identification. Reference [11] proposed a tunnel centerline and section extraction method based on fractional calculus, 3D invariant moments, and best-fit ellipse for tunnel monitoring. This method studies a new smoothing template for TLS point cloud denoising, and on this basis, we propose a new method for extracting the tunnel central axis based on 3D invariant moments. However, in the aspect of building deformation extraction based on points [12], the application of TLS to building monitoring in mining areas is still in the manual stage [13], [14]. Analysts usually assume artificial signs as key points, manually select key points in single phase point clouds, and manually determine the relationship between the key points of two distinct phase point clouds. When extracting the deformation of buildings in the mining area through the above methods, the following limitations are encountered: (1) The layout of artificial signs requires considerable manpower and material resources. (2) During the monitoring process, it difficult to avoid the separation and loss of artificial signs. (3) The accuracy and reliability of the final results depend mainly on the analysts' experience and workload. (4) The manually extracted deformations is subjective, and it is unreasonable to use its results as a standard. Therefore, it is necessary to extract the deformation of buildings in mining areas based on TLS point clouds with minimal manual intervention by taking nonartificial signs as key points.
With this background, an automatic deformation extraction method for buildings in mining areas based on TLS point clouds is proposed in this paper. This method considers the corner points of doors and windows on the wall as key points and realizes automatic extraction of deformations. Section 2 introduces the related concepts and the implementation of this method. Section 3 describes the experiments and applications. Finally, the discussion and conclusion are presented in Sections 4 and 5, respectively.

A. OVERVIEW
The automatic extraction method in this study considers the corner points of doors and windows on the building wall as the key points and supports the following two types of input data. (1):On the single-sided wall of two phases, the same area containing key points is selected, and the point cloud in the area is taken as the input data. (2): The single-sided wall point cloud of the two phases is directly taken as the input data. Based on the wall scanning point cloud, the main steps of deformation extraction include data preprocessing, key point extraction, and deformation value calculation (Fig. 1).

1) ACQUISITION OF 2D BOUNDARY POINT CLOUD
For the building walls in the mining area, the corner points of the doors and windows are always located on the boundary line. Compared to 3D space, line fitting on the 2D plane is simpler, and thus we obtain a 2D boundary point cloud.
First, the wall plane is fitted using the random sampling consistency algorithm (RANSAC) [15], and the wall point cloud is extracted based on the point index. Next, the point cloud boundary estimation method based on normals [16] is used to obtain the boundary point cloud of the wall, and the boundary point cloud is projected onto a coordinate plane at a small angle to the wall plane ( Fig. 2(a)). Finally, the boundary points located in the R neighborhood of the bounding box are deleted based on the 2D bounding box of the boundary point cloud ( Fig. 2(b)).

2) DENOISING OF 2D BOUNDARY POINT CLOUD
The 2D boundary point cloud obtained in Section 2.2.1 consists of two parts: boundary points and noise points. Owing to the simple structure of the building wall in the mining area, the boundary lines are either parallel or perpendicular to each other. After obtaining two mutually perpendicular boundary lines, the slope of the distance line from the boundary point to the boundary line became close to 0. Based on this characteristic, the boundary points and noise points were distinguished.
First, on the 2D projection plane, two mutually perpendicular boundary lines were fitted using RANSAC ( Fig. 2(c)) [17]. Second, the distance from the points in the 2D boundary point cloud to the reference lines was calculated in turn and sorted according to the distance ( Fig.  2(d)). Third, in the sorted point set [p1, … , pi, pi+1, …, pj, … ], the slope of each point on the distance line was calculated ( Fig. 2(e)). Assuming the distance difference between points pi and pj as the temporary slope, to reduce the influence of errors, the slope of point pi is expressed as the temporary slope mean of all the points in the K neighborhood. Finally, based on the given slope threshold, the noise points of the boundary point cloud are removed after deleting the points whose slope exceeded the threshold.

C. KEY POINTS EXTRACTION
Key point extraction is crucial to deformation extraction. We extracted the key points on the projection plane based on the boundary point cloud obtained by data preprocessing. The process is expounded based on three aspects: the splitting of boundary lines, the clustering of seed key points, and the judgment of key points.

1) SPLIT OF BOUNDARY LINES
Key points are represented as intersections of boundary lines, yet not all intersections are key points. These intersections, without any processing or judgment, are called seed key points. Boundary line splitting is used to obtain the seed key points.
Before splitting, we set the termination condition as the number of remaining points in the boundary point cloud becoming less than the threshold number. As the noise in the boundary point cloud cannot be completely removed, the threshold number is usually not zero. When splitting, a wall boundary line was fitted based on RANSAC, and its model parameters were recorded ( Fig. 3(a)). Then, we deleted the boundary points on the boundary line from the boundary point cloud according to the point index to complete a pointcloud update. Finally, the above process was repeated until termination conditions were met.

2) CLUSTERING OF SEED KEY POINTS
After splitting, we calculated the coordinates of the seed key points on the projection plane according to the model parameters of the boundary line ( Fig. 3(b)). However, a boundary line may be split into multiple lines, resulting in multiple points in the region where there actually should be only one seed key point. Therefore, we cluster the seed key points.
Before clustering, the termination condition was set, and the number of remaining points in the cluster was 0. During clustering, a seed key point was randomly selected. We assumed the coordinate mean of the seed key points in its R neighborhood as the current clustering center (Fig. 3(c)). Then we deleted the seed key points in its neighborhood to update the cluster set once. Finally, we repeated the above process until the termination conditions were met.

3) JUDGMENT OF KEY POINTS
After clustering, the seed key points in R neighborhood disappear, and the clustering centers become new seed key points ( Fig. 3(d)). Seed key points are divided into two categories: key and non-key points. When judging, the coordinate difference of the boundary points in the neighborhood in the X and Y directions were calculated. In the neighborhood of key points, the two coordinate differences were greater than the threshold (Fig. 3(e)). In the neighborhood of non-key points, only on coordinate difference was greater than the threshold difference ( Fig.  3(f)). Finally, based on this characteristic, after traversing the seed key points, the key points were identified.

D. DEFORMATION EXTRACTION
In Section 2.3, the key points on the projection plane were extracted. To extract the deformation in 3D space, it is necessary to retrieve the 3D coordinates of the key points and establish the key point pair relationship between the two scans.

1) 3D COORDINATE RETRIEVAL OF KEY POINTS
We calculated the coordinates of the key points on the projection plane. Owing to the irregular deformation during mining, we do not obtain 3D coordinates of the key points in the absolute coordinate system by "calculating the rotation transformation matrix of the boundary point cloud before and after projection and realizing the coordinate transformation with the rotation transformation matrix" [18]. When retrieving 3D coordinates, based on the preprocessed boundary point cloud, we determine the boundary point closest to the key point on the projection plane. Then, according to the point index, its 3D coordinates are allocated to the key points to complete the 3D coordinate retrieval of the key points ( Fig. 4(a)).

2) RELATIONSHIP ESTABLISHMENT OF KEY POINTS
When establishing this relationship, the global and local registrations between the input point clouds of the two phases are subjected to normal distribution transformation (DNT) and iterative nearest point algorithm (ICP) [19]- [21]. Then, based on the registered input point cloud, the closest key points of the two phases are considered as the key points with the same name, so as to complete the relationship establishment between the key points of the two phases ( Fig.  4(b)).
It is worth mentioning that the above registration is only to establish a point-to-point relationship. After the relationship is established, the point cloud whose position changes in registration will immediately restore the position before registration.

3) CALCULATION OF DEFORMATION VALUE
The 3D coordinates and pair relationship of the key points obtained above can be expressed as follows. (1) Where, ΔX is the horizontal deformation of X direction, ΔY is the horizontal deformation of Y direction, ΔW is the subsidence deformation.

1) EXPERIMENT SITE
The experimental site was the experimental base of the Anhui University of Science and Technology in China. For the building wall shown in Fig. 5(a), we conducted five phase scans at the same location. After each phase scan, the TLS was not moved to ensure that the point cloud data of the five phase scans were in the same coordinate system.
HI-TARGET HS-650 laser scanner was the TLS used for data acquisition (Fig. 5(b)). The scanner uses a pulse measurement method for ranging and provides two laser pulse emission frequencies: 300 kHz (indoor) and 100 kHz (outdoor). The higher the frequency, the shorter the measurement distance. The ranging range is 1.5-650 m, the scanning range is 0-360 ° horizontally and -40-60 ° vertically, and the ranging accuracy of 100 m can reach 5 mm.

2) EXPERIMENT RESULT
In the simulation experiment, the point cloud of the scanning wall was assumed as the input data, and the automatic extraction method was used to extract the deformation between the first phase and the remaining phases. The deformation values of the key points, namely the movement ΔX, ΔY in the X and Y directions, respectively, and subsidence ΔW were acquired. The results are shown in Table I . It can be seen from Table I that most of the deformation values of the key points are distributed between -5 and 5 mm, and only a few exceeded ±5 mm; however, no value exceeded ±9 mm, which is not considerably different from the real value of 0 mm. In the simulation experiment, four key points in each phase of the point cloud data were identified, and the relationship between the key points of the two phases was established accurately. Overall, this method demonstrates high accuracy in extracting the deformation value of buildings in mining areas.

1) APPLICATION AREA
The application area of this study was Jianglou village, Bozhou City, China ( Fig. 6(a)). Below the village, a mining working face with a strike length of approximately 1000 m and dip length of approximately 300 m is mined (Fig. 6(b)). In this village, we selected two buildings and used TLS to scan the wall with the most key points (Fig. 6(c)).
The HI-TARGET HS-650 laser scanner was used to collect the data. Four target balls were placed near the TLS for coordinate conversion to obtain the absolute coordinates of the scanning point cloud. The first scanning date was March 28, 2021, and the mining working face had not been mined at this time. The second scanning date was July 28, 2021. At this time, the mining working face was pushed below the village.

2) APPLICATION RESULT
When extracting the deformation between scans of the two phases, building A took the point cloud in the red rectangle as the input data ( Fig. 6(d)), and building B directly took the scanned point cloud as the input data (Fig. 6(e)). The deformation value of key points was also given, and the absolute error (AE) between the automatic extraction and manual extraction results was calculated. In manual extraction, to reduce the influence of human error, each key point in each phase was extracted four times. The mean value of the four extracted coordinates was assumed as the key point coordinates, and the deformation was calculated. The results are shown in Table II . It can be seen from Table II that AE is mostly distributed between 3 mm and 7 mm, and not greater than 9 mm. The deformation values obtained by the automatic and manual extraction methods were almost equal. In the two buildings observed in this study, Building A moved to the northwest along with subsidence. The maximum movement of the building to the northwest was 154 mm; the maximum movements to the west and north were 95 mm and 59 mm, respectively, and the maximum subsidence was 113 mm. Building B moved horizontally and sunk less. The maximum movement of the building to the northwest was 102 mm, and the maximum movements to the west and north were 69 mm and 74 mm, respectively.
Combined with the positional relationship between the building and the mining working face, it was found that compared to Building B, Building A was closer to the current goaf center of the mining working face and must experience greater horizontal movement and subsidence, which is consistent with the extracted deformation (Table Ⅱ) and the measured point clouds (Fig. 7). However, there was little difference in the movement values of buildings in the Y direction, and buildings A and B did not move to the south. This was mainly caused by the following three factors: (1) Until the end of phase II scanning, the mining working face had not been mined below the village. (2) Compared to the strike of the mining working face, the buildings along the dip were less affected by mining. (3) The buildings were affected by working face mining in the north of the village.

A. ADVANTAGES OF DISTANCE SLOPE FILTER
Without considering the influence of instrument error, the noise points in the 2D boundary point cloud obtained in Section 2.2.1 can be divided into two types. Type (1): When estimating the boundary point cloud of the wall based on normal, the boundary points that should not exist appear because of unreasonable parameter selection. Type (2): Owing to the influence of object occlusion, holes are often present in the scanning point cloud, resulting in boundary points that should not exist. Therefore, in data preprocessing, a noise removal filter based on slope of distance line (distance slope filter) is proposed in this paper.
Considering the building B studied in engineering applications as an example, based on the 2D boundary point cloud obtained in Section 2.2.1, the distance slope filter and traditional noise removal filters (radius filter and statistical filter) were used for denoising [22]. The results are presented in Fig. 8. Comparing the results before and after noise removal, there was no obvious loss in the boundary point cloud after denoising by all three filters. For noise points of type (1), the above three filters could remove them (red circle). For the noise points of type (2), only the distance slope filter could remove them (black circle). In engineering applications, owing to the complex environment in the mining area, it is difficult to avoid the emergence of noise points of type (2). Therefore, the distance slope filter greatly improved the applicability of the automatic extraction.

B. STABILITY OF KEY POINTS EXTRACTION
At present, when TLS is applied to building deformation monitoring in mining areas, manual extraction of key point coordinates is yet the most commonly used method, and the manual extraction results are used as standard results. The accuracy of the automatic extraction method was verified through simulation experiment and engineering application. Therefore, considering building B as an example in engineering applications, the stability of the key point extraction was verified. When extracting the coordinates of the key points, we extracted the coordinates four times using the automatic extraction method and the manual extraction method, calculated RMSE of the coordinates, and assumed RMSE as the index to judge the stability.
As can be seen from Table III, except that the RMSE value of key point 3 in the X direction is 4.1 mm, the RMSE value acquired by the automatic extraction method did not exceed 2.0 mm, and most of them were distributed in the range of 0-1.5 mm. The RMSE value acquired by manual extraction method does not exceed 7.1 mm, and most of them were distributed in the range of 0.5-5.0 mm. From the distribution of RMSE values and their upper and lower limits, it can be seen that the automatic extraction method has higher stability in key point extraction as compared to the manual extraction method; therefore, considering automatic extraction results to be the standard is more reasonable.

Ⅴ. CONCLUSION
This study proposed an automatic deformation extraction method for buildings in mining areas based on TLS point clouds. The automatic extraction method takes the corner points of doors and windows on the wall as the key points and realizes the automatic extraction of deformation. The main conclusions of this study are as follows: (1) The accuracy of the automatic extraction method in mining area building deformation extraction was verified. There is negligible difference between the deformation values extracted by this method and the real values. Most of the errors were distributed between -5 and 5 mm with only a few exceeding ± 5 mm; however, no error exceeded ± 9 mm.
(2) The automatic extraction method includes a denoising filter for the wall boundary point cloud, i.e., a distance slope filter. Compared to traditional denoising filters, such as statistical and radius filters, noise points were removed more effectively by the distance slope filter. Therefore, the applicability of this method has been greatly improved in the complex measurement environment of mining areas.
(3) For the key points of the same building walls, automatic method and manual method were used to extrac, respectively. The RMSE values acquired by this method did not exceed 2.0 mm, and most of them were distributed in the range of 0-1.5 mm. The RMSE values acquired by manual method did not exceed 7.1 mm, and most of them are distributed in 0.5-5.0 mm. Compared to the manual extraction method, this method shows greater stability.
(4) This method can avoid the problems of timeconsuming and laborious layout of artificial signs and loss of artificial signs in the process of monitoring. Simultaneously, it can avoid the problem of subjectivity as the manual extraction of the key points mainly depends on the experience and workload of the analysts, and hence the results are not standard. This method shows good potential for building deformation monitoring in mining areas.