Road Damage Detection From Post-Disaster High-Resolution Remote Sensing Images Based on TLD Framework

Road is one of important traffic lifelines that could be damaged after disaster by landslide rubble, buildings debris, and collapsed branches of trees. Therefore, road damage detection and assessment using post-Disaster High-Resolution Remote Sensing Images is extremely important for finding optimal paths and conducting rescue missions. In an emergency context, the existing methods based on change detection for road damage detection are difficult to achieve due to the mismatch of different data sources, especially for rural areas where the pre-disaster remote sensing imagery are hard to obtain. In this paper, a novel method based on the Tracking, Learning, and Detector (TLD) framework for detecting the damaged road region from post-disaster high-resolution remote sensing image is presented. First, a spoke wheel operator is employed to define the initial template of road. Then, the TLD framework is used to identify the suspected road damaged areas. Finally, the damaged road areas are extracted by pruning the false damaged roads. The proposed method was evaluated using post-disaster high-resolution remote sensing images collected over Beichuan, China in 2008 and Lushan, China in 2013. The results show that the proposed method is feasible and effective for road damage detection and assessment. Our main conclusion is that such an approach qualifies for practical use.


I. INTRODUCTION
Earthquakes, floods, and wildfires, cause extensive destruction to infrastructure, flatten buildings, and block roads, which results in heavy economy and social losses. Roads are a lifeline. After a disaster occurs, road damage detection and assessment are the foundation for emergency response actions and rescue work. Various kinds of remote sensing data, such as aerial or satellite images, Lidar and SAR, have been widely used to identify, detect, and assess road damage for disaster [1]- [5]. Specifically, high-resolution aerial images can be obtained in a controlled fashion, both in terms of time and flight planning and at much higher geometric, spectral, and radiometric resolution to meet emergency The associate editor coordinating the review of this manuscript and approving it for publication was Wenming Cao . response needs. This is more suitable for fast and reliable post-disaster damage assessment due to accessibility and rapid acquisition, including the identification of damaged road area. Detecting damaged road region using highresolution aerial images can support faster and more effective decision-making and disaster management.
Many approaches for detection and estimation of road damage by earthquakes have been proposed. These approaches can be categorized into three types. The visual interpretation methods [6]- [9] are employed to detect and assess the road damage using various remote sensing images and GIS data, but depends on many auxiliary tools (e.g., ArcGIS). Visual interpretation is the most widely used in practice for road damage detection, as currently it is the most accurate method. Unfortunately, it is timeconsuming and requires trained operators, and thus not appropriate for rapid damage assessment. Change detection method [10]- [12], compare various pre-and post-disaster features, can lead to more accurate and reliable results, including the height feature changes from Lidar and stereo images, texture feature change from optical images and the backscattering, and correlation coefficients and coherence from SAR images. However, multi-temporal techniques have a major limitation in that many areas, especially in developing countries, do not have pre-disaster images. The match between pre-and post-disaster is another challenging issue. The single post-disaster image method is another way to detect road damage using only one set of remote sensing data collected after a seismic event [13]- [15]. Post-disaster image methods are most suited for creating a very detailed inventory, such as satellite, aerial and SAR images. This approach mainly uses feature extraction supported by auxiliary pre-earthquake GIS vector data to detect road damaged area. This method can also be used detect the extent of road damage in real time. Given that its speed and automation, this approach is the most commonly employed technique for damaged road detection. Therefore, this method is more suitable for detecting road damaged region for emergency response actions and rescue work.
Based upon our initial work, in this paper, the motivation for our method based on the Tracking Learning and Detector(TLD) framework is to detect the damaged road regions. TLD framework was originally used to track the objects in a video stream, which includes three sub-tasks: tracking, learning and detection. The tracker follows the object from frame to frame. The detector localizes all appearances that have been observed so far and corrects the tracker if necessary. The learning estimates detector's errors and updates it to avoid these errors in the future [16]. Each sub-task is addressed by a single component and the components operate simultaneously. Since road tracking is similar to object tracking in a video, we utilize TLD framework to detect undamaged road regions. The surfaces of road vary in appearances due to the influence of climate and natural corrosion. In order to improve the stability of road detection, we train a classifier with an iterative P-N learning strategy to discern these differences. In the case of streaming video, detector cannot locate the object if the object disappears from the video stream, but the object can be re-located by the detector once they reappear. Similarly, a road cannot be detected and tracked if road regions are damaged, but the undamaged road regions can be detected once reappearing in the road tracking. Therefore, the following strategies are employed to detect damaged road regions: (1) The spoke and wheel are used to descript the initial road template; (2) The Tracking, Learning and Detector framework is employed to obtain the suspected damaged area of the road; (3) We prune the false damaged road based on the color invariant. In experiments, the proposed method is tested on high-resolution remote sensing images from different scenes. The results will show that the proposed method improves the precision and stability of road damaged detection.
The main contribution of our work is summarized as follows: (1) We analyze and summarize the challenges of road damaged detection with respect to the prevailing Visual interpretation method, change detection method and the post-disaster image method.
(2) To address these issues, a novel road damaged detection framework based on TLD from post-disaster remote sensing images is proposed in this paper. In our framework, the road detector is self-adaption and have subjective factors since the learning model is constantly being updated to detect the new road samples, which avoids detection errors since the surfaces of road vary in appearances due to the influence of climate and natural corrosion.
(3) We utilize color invariant to prune the false damaged road due to the shadow and vegetation, which improves the accuracy of road damaged detection.
(4) We experiment on three remote sensing datasets to show that our framework achieves state-of-the art results on post-disaster remote sensing images quantitatively, qualitatively, and efficiently.
The rest of the paper is organized as follows. Section II presents our proposed a method to detect the damaged road area based on TLD framework from Post-Disaster High-Resolution Remote Sensing Images. Section III details the extensive experiments and results of the proposed method for road damage detection. Section IV presents some conclusions.

II. METHODOLOGY
In this study, a novel method of road damaged detection from post-disaster high-resolution remote sensing images based on TLD framework is proposed. The proposed method has three successive steps. First, initial location of road is manually set by one or more valid seed points, a spoke wheel operator is used to generate the initial road template according to these seed point. Second, the suspected damaged area of the road is extracted based on TLD framework. Finally, the false damaged regions of the road are corrected based on color invariant. The general framework of the proposed method is shown in Figure 1.

A. THE DEFINITION OF INITIAL ROAD TEMPLATE
In our paper, we consider the problem of tracking an arbitrary road in remote sensing image, where initial location of road is manually set by one or more valid seed points. The initial road template is generated according to the starting seed point. In remote sensing image, roads are thin and long linear structures with the following geometric and radiometric features: (1) The width of road is constant during a distance, and roads are thin elongated structures with a bounded width.
(2) The local area of a road pixel is a nearly homogeneous region, which is anisotropic and directionally rectangular. The advantage of spoke wheel operator is that they can transfer a 2-D local image intensity distribution into a set of 1-D intensity functions. This makes it easier to find the directions of anisotropic structures in an image by comparing the difference among the directional functions. Therefore, we use a spoke wheel operator to describe the road template [17], [18].
A spoke is a line segment with a length of m pixels. There is a sequence of spokes S i (ϕ i , m)(i = 0, . . . , 4n − 1) with common seed point P and evenly spaced angles ϕ i = πi/2n. The pixels set in a spoke wheel W centered at the pixel P with 4n spokes is denoted by W (P,n, m).
The intersection between the road boundary and the spoke is used to determine a local homogeneous region around a pixel. However, the location of the road boundary cannot be obtained in advance. In order to find the intersection of a spoke and the boundary of a road, we start from p and move in the direction of the spoke, and observe the absolute intensity differences between p and the pixels along the spoke. The differences are small when the pixels are near to p; but they may become larger when the pixels are far away from p. Let S i be the ith spoke at pixel p. The cutting point, denoted by E i , on S i is the first pixel such that where δ(W (p, n, m)) is the intensity standard deviation on W(p, n, m).Notice that δ(W (p 1 , n, m)) = δ(W (p 2 , n, m)) usually holds if p 1 = p 2 .Therefore, the thresholding in (1) is adaptive. The road associated with each pixel is an anisotropic structure; that is, the distance E i − p in some directions is much longer than that in other directions. To find the road directions, we connect the cutting points on all spokes around a pixel p in counterclockwise direction, which results in a closed polygon. The minimal bounding box of the closed polygon represents the road template of the pixel p, denoted by F(p). Figure 2(b) shows a road template of the pixel p in Figure 2(a).

B. EXTRACTING THE SUSPECTED DAMAGED REGION
This section investigates the suspected damaged road areas detection based on the TLD framework. The key idea of our method is that the TLD framework is used to detect the undamaged road regions, but road regions are suspected to be damaged if road cannot be detected and tracked. The proposed method is characterized as follows. First, road tracking refers to the task of estimating a path and forms the road network from the initial road template; the tracker is likely to fail and never recover if road regions is damaged. Second, road detector localizes all appearances of road regions based on LBP feature. Finally, learning based on P-N observes performance of tracker and detector, estimates detector's errors and generates training examples to avoid these errors in the future. The learning component assumes that both the tracker and the detector can fail. By the virtue of the learning, the detector generalizes to more object appearances and discriminates against background.

1) ROAD TRACKING
Road tracking refers to the task of estimating a path and forming a road network from the initial road template in an iterative line growing process along the road direction [14].
The key idea of road tracking is as follows in this paper. The next road region is searched clockwise around the center point P 0 . Our searching starts from the perpendicular to the current direction of the road and finishes at its opposite direction. There is a transformation relation between the center points of the adjacent road template. The transformation relation is as follows: where the point P 0 (x 0 ,y 0 ) is the center point of the current road template, the point P 1 (x 1 ,y 1 ) is the center point of the next road template. l is the step length between two center points. θ is the main direction of road path. θ t is the offset of the main road direction. The rectangular region with the center point P 1 is the suspected road region. In this process, we iteratively detect the feature of road surface in the road network to decide whether to be damaged. Template tracking of road is the most straightforward approach in that case. The road region is described by a target template (an image patch) and its tracking is defined as a transformation that minimizes mismatch between the target template and the candidate patch.

2) ROAD DETECTION
Our goal is to detect the road region from the suspected damaged road region using the aforementioned method. Road detection methods are typically based on the local image features. The feature-based approaches usually follow the processing steps: x feature extraction, y feature recognition, and z model fitting [19]. The undamaged road regions have two obvious characteristics: x The surfaces of road vary in appearances due to the influence of climate and natural corrosion. y The local area of road is anisotropic and directionally rectangular, and each road pixel is within a locally homogeneous region. LBP (Local Binary Patterns) [20], [21] is a robust descriptor that summarizes texture information, and invariant to illumination translation and scaling. Therefore, LBP is suitable to describe the feature of road area. Figure 3 illustrates the LBP workflow. x The neighborhood (N ), usually defined by an angular resolution (typically 8 sampling angles) and radius r of the neighborhoods.
y The binarization function b(x c , x i ) ∈ (0, 1) allows the comparison between the reference point (central pixel) and each one of the points x i in the neighborhood. LBP is applicable when x c (and x i ) are in an ordered set, with b(x c , The output of the LBP at each center position x c is the code resulting from the comparison of the value x c with each of the x i in the neighborhood, with i∈N(x c ). The LBP codes can be represented by their numerical value as formally defined in (5).
LBP codes can take 2 |N | different values. In predictive tasks, for typical choices of angular resolution, LBP codes are compactly summarized into a histogram with 2 |N | bins, being this the feature vector representing the road region. Based on the features, a RF(random forest) [22], [23] classifier is used to train these samples that were manually collected as road and non-road regions. Given a set of labeled training samples, RF can learn a nonlinear decision boundary to discriminate between the two classes.
However, if road is damaged, we cannot detect the road area using the aforementioned method unless we expand the search area. As shown in Figure 4, for each step length l, we will detect the road area from 0 to 180 0 . If road region cannot be detected, we increase the step length l for detecting the road area. And so on, until we find the road area. The area between starting point A and ending point B is the road damaged area.

3) ROAD SAMPLES LEARNING BASED ON P-N
Since we can discover variation in the appearances of the same road, the single classifier and detector method is not suitable for road detection. The P-N learning as a semisupervised learning method can iteratively learn and update the classifier on the labeled and unlabeled samples, as road sample learning guarantees improvement of the detector. As shown in Figure 5, we utilized RF(random-forest) and P-N learning to detect the roads from remote sensing images.
The key idea of P-N learning analyzes road by two types of ''experts'': x P-expert -recognizes missed detections, and y N-expert -recognizes false alarms. The estimated errors augment a training set of the detector, and the detector is retrained to avoid these errors in the future. Both of the experts make errors themselves, however, their independence enables mutual compensation of their errors. Road tracking is seen as an iterative line growing process, adjacent samples of road track are considered as positive samples, while samples far away from the road trajectory are considered as negative samples. Positive constraints are used to find the unlabeled data on the road trajectory, negative constraints are used to VOLUME 10, 2022 distinguish roads from complex background. P-N learning is described in detail as following.
Let L l = {(x, y)} be a labeled set for examples, x is an example from a feature-space F and y is a label from a space of labels Y = {−1,+1}. L u is an unlabeled set of examples. Both labeled L l and unlabeled L u are input into the P-N learning, where l u. The task of P-N learning is to learn a classifier h: F -> Y from labeled set L l and bootstrap its performance by the unlabeled set L u . Classifier h is a function from a family H parameterized by θ. The P-N learning consists of four blocks: x Initial Classifier. The initial classifier h is trained by the labeled training set L l that were manually collected as road and non-road regions. The training set is then passed to supervise learning which trains a classifier, i.e. estimates the initial parameters θ 0 .
y Online Learning. In the process of road tracking, the road and background region are detected continuously. These regions are input to the classifier for incremental learning.
z Classifier Correction. The learning process then proceeds by iterative bootstrapping. In iteration k, the classifier trained in previous iteration classifies the entire unlabeled set, { These examples are added with changed labels to the training set. The iteration finishes by retraining the classifier, i.e. estimation of θ k . The process iterates until convergence or exceeds the number of iterations.

C. PRUNING THE FALSE DAMAGED ROADS
Damaged road detection would be misjudged due to the cover of shadows or vegetation. In order to prune false damaged roads, we also adopt color invariance [24] to detect the vegetation regions using the following equation: Pixels with a color index larger than the threshold identified by Otsu's method are marked as vegetation or shadows. However, shadow and vegetation detection are not always accurate no matter what algorithm is applied. This does not affect our final detection result as the main reason for this is that emergency managers need to know whether the vehicles can pass over the road for emergency response actions and rescue work.

III. EXPERIMENTAL RESULTS AND DISCUSSION
The experimental results and used datasets are described in this section. To evaluate the quality of damaged road region detection, we conducted qualitative, quantitative and efficiency experiments to evaluate and test the performance of the proposed method for road damaged detection. A visual inspection of damaged detection images subjectively reveals signs of clarity and completeness. An objective, empirical evaluation using the precision, recall, and the detection runtime for an image tested the proposed method in relation to other methods. Precision and recall were employed for in the objective empirical evaluation. These functions are defined as: Recall = TP TP+FN (8) where TP is the damaged road pixels extracted by the proposed method which are consistent with the reference data, FN is the damaged road pixels which are in the reference data but not in the extracted result, and FP is the extracted damaged road pixels which are not in the reference data. In addition, parameter selection and sensitivity analysis were used to further assess the practicability of the proposed method. Experiments were carried out on a PC with an 8-GB memory and an Intel Xeon CPU E3-1220 with 3.10 GHz on MATLAB 2019.

A. STUDY AREA AND DATA SOURCES
The experimental data were two groups of high-resolution aerial images provided by the national disaster reduction center of China for the urban areas of Beichuan and Lushan city, Sichuan province, China. As depicted in Table 1, the different locations, dates and sensors were included in each group. This resulted in a large variability in image quality, in the environmental conditions and typology of road damage depicted in the images. The Resolution of images also varies within each group.

1) QUALITATIVE EVALUATION
In order to verify the damaged detection effect, the detection effect of our method is measured by the visual images after  detecting. The performance of the proposed method is illustrated through the testing result of the site of the Beichuan and Lushan earthquake ruins as an example. The results of damaged detection roads are shown in Figure 6, Figure 7 and Figure 8. As shown in Figure 6, there are experimental results of damaged road region detection from the datum(Lushan1, Sichuan province, China). Figure 6 (a) shows the inscribed line of road extracted manually from remote sensing images, Figure 6 (b) shows that some suspected damaged road regions are extracted from remote sensing images, Figure 6 (c) shows that damaged road regions are extracted from remote sensing images after removing the false damaged road regions due to regions blocked by the vegetation. Road is covered by vegetation with obvious shadows, the result show that our method can remove the false damaged road regions because of the blocked area by the vegetation.
As shown in Figure 7, there are experimental results of damaged road region detection from the datum(Lushan1, Sichuan province, China). Figure 7 (a) shows the inscribed line of road extracted manually from remote sensing images, Figure 7 (b) shows that some suspected damaged road regions are extracted from remote sensing images, Figure 7 (c) shows that damaged road regions are extracted from remote sensing images after removing the fake damaged road regions due to the blocked regions by the vegetation. However, a few falsely detected damaged regions still exist in some regions. For example, Figure 7(c) shows that the road regions are damaged by landslides, but these regions are surrounded by vegetation, our method determining this case as undamaged regions of road, so this case cannot be detected.
As shown in Figure 8, there are experimental results of damaged road region detection from the datum(Beichuan, Sichuan province, China). Figure 8 (a) shows the inscribed line of road extraction manually from remote sensing images, Figure 8 (b) shows that some suspected damaged road regions are extracted from remote sensing images, Figure 8(c) shows that damaged road regions are extracted from remote sensing images after removing the fake damaged road regions due to the blocked regions by the vegetation. However, some few falsely detected damaged regions still exist in some regions. For example, Figure 8 (c) shows that existing debris or rubble is on the road surface because the road was damaged in the earthquake, but these road's regions were only slightly damaged, and cars can still pass on these roads. Our method determining this case as undamaged regions of road, so slightly damaged road regions cannot be detected.

2) QUANTITATIVE EVALUATION
To further evaluate the performance of our method, we utilized two indices to describe the performance of the proposed method. Precision and recall were compared to methods based on the gray-level co-occurrence matrix (GLCM) [25], the HOG feature [26], the SURF feature [27] and LBP feature. It can be seen from Table 2 that the detection results are better precision and recall than all alternate features with undamaged road regions. Since the undamaged road regions are flatten and the texture of its surface varies less, the LBP feature is more suitable for describing road texture information than HOG, GLCM and SURF, suggesting that the proposed method can obtain better results.
We also compared the proposed method to damaged road region detection with other methods. The study tested the proposed method against the OCSVM method in Reference [10], the GA and SVM Classification method in Reference [11], and the knowledge-based detection method [14]. Table 3 shows the accuracy results compared to the other methods. Compared to the traditional supervise learning method [10], [11], the Reference [14] utilizes knowledge base to detect the damaged road regions, knowledgebased method for road damage detection, which achieves high accuracy. Both Reference [10] and Reference [11] use feature and SVM classifier to detect the damaged road area, single datasets are used to train their classifier, but the surface of road is variant due to the influence of climate and natural corrosion, hence their classifier could not distinguish the damaged and undamaged area for road. The Reference [14] method does not require the pre-disaster data and achieves high accuracies, but undamaged region may falsely be classified as the damaged region due to the influence of vegetation and shadow. As shown from Table 3, the proposed approach can achieve the best precision and recall results than the competing methods. Our method outperforms all the competitive methods on datasets since Our approach takes into account changes in the road surface and the fake damaged road detection due to vegetation and shadow. Not only that, in the next section we will discuss the advances of our method in terms of operational efficiency.

3) OPERATING EFFICIENCY
In addition to precision assessment, another important aspect for road damage detection method is the testing speed. Table 4 shows the run times of different methods for road damage detection. Since Reference [10], Reference [11], Reference [14] and our method are well suited for parallel computation on GPU, we also give the corresponding run times on GPU. As in [28], we do not count the memory transfer time between CPU and GPU.
As illustrated in Table 4, we can see that the proposed method can have a relatively high speed on CPU and GPU, its running time is very competing in contrast to other existing methods. Further, the proposed method does not require  the pre-disaster data and has the smaller complexity. Taking detection performance and flexibility into consideration, our method is very competitive for practical applications.

4) SENSITIVITY ANALYSIS
In the proposed road damage detection method, the step length l in Figure 9 is a key parameter and need be manually adjusted, which affects the detection accuracy. In our experiments, we tested that the different values of l were obtained the detection precision and recall, which were used to evaluate the sensitivity of l to the detection results. The experimental results of the datum are shown in Figure 9.
The highest overall quality for datum was achieved when l was 25. The detection precision and recall tended to be stable on this datum when l was 15-30, but the detection precision and recall began to show a gradual decrease when l was greater than 30. As shown in Figure 9, if the step length l is greater than the length of damaged road, the template of road   detection will skip to the damaged road region. Therefore, we can't detect damaged roads when l is bigger and bigger. According to this evaluation, the l set should have a value in the range of (15,30). In this way, 25 is selected as the threshold value in this paper.
The proposed method detects the heavy damaged road region rather than the slight damaged road regions. The main reason for this is that emergency managers need to know the damaged region of road for emergency response actions and rescue work, but the slight damaged road regions do not affect the passage of vehicles. Therefore, the proposed method is an effective method to detect the damaged road regions in both the qualitative and quantitative aspects.

IV. CONCLUSION
In this paper, we have presented a novel method for damaged road region detection based on the TLD framework, which can support emergency response actions and rescue work after a disaster. Firstly, we use a spoke and wheel metaphor to describe the road template. Then, the tracking-learningdetection framework is used to extract the suspected damaged region of the road. Finally, the damaged road areas are extracted by pruning the false damaged roads. The experimental results show that the proposed method is feasible and effective in detecting the damaged road regions, and demonstrate a practical significance to the study of techniques for damaged road detection.
The proposed damaged road region detection method based on road tracking detection has great potential in road damage detection in emergency response actions and rescue work after disasters. In the future, the proposed method will be adapted to assess the road damage level assessment.
KANG ZHAO received the Ph.D. degree in geographic information system from Wuhan University, Wuhan, China, in 2021. He worked as the Director of the Yunnan Provincial Geomatics Centre, Information Development Department, China. He worked in Yunnan geological environment big data management and analysis platform. He is a member of the Chinese Society of Geodesy, Photogrammetry, and Cartography. He has authored several papers. His research interests include geographic computing, geographic information systems, and spatiotemporal big data. JIHUI TU received the Ph.D. degree in photogrammetry and remote sensing from Wuhan University, Wuhan, China, in 2017. He is currently an Associate Professor with the Electronics and Information School, Yangtze University, Jingzhou, China. His research interests include deep learning, computer vision, and remote sensing image processing. VOLUME 10, 2022