Sewer Pipeline Fault Identification Using Anomaly Detection Algorithms on Video Sequences

Most existing sewer pipeline condition assessment methods determine the presence and types of faults via examination of videos, which is a time-consuming and labor-intensive process. A few automatic methods based on image processing techniques can be used to detect specific faults. However, these methods have limitations due to the presence of unpredictable sewer pipeline fault patterns. Deep learning methods have also been applied to sewer pipeline fault detection. However, these methods require a large amount of annotated data to obtain reliable results. In this paper, we propose a fault detection method that applies unsupervised machine learning based anomaly detection algorithms with feature extraction to videos recorded by new sewer pipeline visual inspection equipment. The recorded videos are regarded as sequence signals, which are converted into feature vectors, followed by application of an anomaly detection algorithm. Unlike existing methods, the proposed method is computationally efficient as it does not require an annotated fault sample database for training fault detection models. We evaluate various anomaly detection algorithms and feature combinations on real sewer pipeline data collected in Shenzhen, with an overall accuracy result of above 90%. The proposed method provides a new and fast technique for surveying urban sewer pipelines, and to facilitate further research in this area, we have made the code and data used in this paper publicly available.


I. INTRODUCTION
An urban underground sewer pipeline system is an important component of public infrastructure, as it plays a major role in maintaining healthy environments. A sewer system can encounter several problems during its operation, such as cracking, misaligned connections, channeling, or silting, which may be caused by the natural environment, engineering construction, or self-aging. The absence of regular sewer pipeline inspection and maintenance can easily cause blockage or serious damage [1]. As a result, this will not only affect the daily rainwater and sewage discharge in a city, but there can also be secondary problems such as water logging and environmental pollution in extreme cases. Therefore, regular The associate editor coordinating the review of this manuscript and approving it for publication was Gulistan Raja . inspection of underground sewer pipelines is essential to avoid sewer pipeline deterioration [2].
Sewer pipelines are placed in underground environments and can develop complex faults. These pipelines are usually monitored using a closed circuit television (CCTV) inspection technique, which records videos and then uses them to assess the structural conditions of sewer pipelines. However, this technique suffers from high costs and low efficiency [3], [4]. Automated defect classification systems based on video surveillance are an important tool that are economical and efficient. However, when applied to complex underground sewer pipeline systems, automated defect detection techniques based on traditional image processing methods have several limitations [5]- [7].
In this paper, a new type of visual inspection equipment is used to record video data of underground sewer pipelines. This equipment is low-cost, lightweight and easy to operate. To the best of our knowledge, this is the first time this equipment has been used for large-scale sewer pipeline inspection applications. The sewer pipeline video data is also unique and has been presented in this paper for the first time.
This paper proposes a sewer pipeline fault detection method based on anomaly detection algorithms applied to the video data recorded by this new equipment. The main idea in this paper is inspired by the human visual recognition mechanism. When an artificial visual interpretation method is used to identify faults from a video, any frame containing faults will be obviously different from the previous and following frames in the video sequence. In addition, the image of the fault constitutes only a small portion of the video sequence and consequently, and it can be regarded as an abnormal or outlier signal.
The proposed method for detecting faults in the videos can greatly improve the efficiency compared to manual inspection and traditional image processing, because it does not require any prior information about the faults. It can also significantly reduce the amount of manually labeled data that is required by supervised learning methods in the training stage.
The novel contributions of this paper are as follows: 1. We extract image features from the image sequence and generate new features by further combining the extracted features. These combinations can help improve the fault detection accuracy.
2. We evaluate several unsupervised anomaly detection methods using combinations of different features as input, which allows identification of the combined set giving the best fault detection accuracy.
3. We carry out evaluations over both large and small image sequences, which shows that our proposed method can be applied in practical scenarios with limited availability of training data. 4. The data used in our paper and the implementation of the proposed method are publicly available on https://github. com/fangxu622/Sewer-Pipeline-Defect-Identification, which can facilitate further research and development in this area. This paper is organized as follows: Section II presents an overview of related work. Section III explains the proposed method and the data used in this paper, while the performance evaluation of the method is presented in Section IV. Section V concludes the paper.

II. RELATED WORK
Research on automatic detection of underground pipeline faults based on image processing technology has practical significance. Initially, conventional computer vision techniques were explored for automated interpretation of underground pipeline video data. However, the application of these techniques required complex feature extractors to be used. For example, Myrans et al. [8] used GIST method to extract features from CCTV image sequences and fed these features to a classifier to detect faults. Yang and Su [5] proposed using morphological operation and segmentation to detect four types of pipeline faults: open joints, cracks, fractures and broken pipes. Halfawy and Hengmeechai [9] proposed an algorithm for automated crack identification using sewer pipeline inspection images obtained from CCTV. First, Sobel operator and Hough transform were used for image preprocessing. Subsequently, morphological operations were applied to enhance candidate crack segments, and customized filters were employed to remove noise edges and extract crack segments.
To take advantage of video sequences available from CCTV data, Guo et al. [10] proposed a change detection based approach for automated fault detection from videos using frame differencing. Hawari et al. [4] developed an automated tool that integrated various image processing and shape analysis techniques to detect and identify different types of faults. The authors showed that cracks, joints and sediments were identified with accuracies of 74%, 65% and 53%, respectively. Yang and Su [2] applied machine learning in a pipeline fault diagnosis system. First, the authors combined the wavelet transform and a co-occurrence matrix to calculate the image texture transformation. Then, using real sewer inspection data, a radial basis network, support vector machine (SVM) and back-propagation ANNs were compared for sewer pipeline fault classification.
The superior performance of deep learning methods in the field of computer vision has led several researchers to apply these methods to sewer fault detection. Kumar et al. [11] proposed multiple binary CNNs for automated defect classification based on CCTV inspection of sewers. Wang and Cheng [12] proposed DilaSeg-CRF integrated dilated convolution and multiscale techniques with RNN layers for automatic severity assessment of sewer pipeline faults. Hassan et al. [13] proposed a sewer fault classification system using CCTV imagery and convolutional neural networks (CNNs). The proposed system showed an accuracy of over 90%. Meijer et al. [14] use CNNs to automatically detect the twelve most common fault types in a dataset of over two million CCTV images.
In recent research literature, different state-of-the-art computer vision based CNN models, such as YOLO, SSD and Faster-RCNN were evaluated in terms of speed and precision for sewer pipeline fault detection using CCTV video data [15]. However, deep learning methods that utilize supervised learning algorithms require many labeled fault samples to train a fault detection model. These methods have two main limitations: 1) The acquisition of manually labeled samples in the training stage incurs a high cost. 2) If the trained model encounters new types of faults or sewer pipeline environments, it may not be able to effectively detect the faults. To improve the speed and accuracy of sewer surveys, assistance from technicians present in the field is necessary. In [16], the authors used a one-class SVM (OC-SVM) algorithm to achieve fault detection on CCTV image data. Unlike the method proposed in [16], we do not use CCTV data and there is no need to train a model using fault and fault-free samples, which makes our method more versatile and practically realizable. Furthermore, our evaluation results show that other anomaly detection algorithms can outperform OC-SVM.

III. METHODOLOGY AND DATA A. METHODOLOGY
In this paper, we combine image processing and anomaly detection algorithms for fault detection using raw video data of sewer pipelines. The video data was taken directly from industrial surveys. First, the video is converted into a sequence of images. These images capture the sewer pipeline states, which can either contain faults or be fault-free, depending on the pipeline being examined. Most of the images do not contain any faults and only a small number of the images correspond to any faults. Due to the characteristics of the data collected by our equipment, which consist of a video stream with a fixed perspective between two manhole covers, the faults indicated on the video can be considered as an abnormal signal within a section of the pipeline. Therefore, we use an anomaly detection method to detect faults based on the video data. We evaluate various anomaly detection algorithms for sewer pipeline fault detection. The flowchart of the proposed method is shown in Fig. 1. The main steps of our proposed method are as follows: 1) The video data is first converted into a sequence of images. 2) Various features are extracted from sequence images, such as local binary patterns (LBP), histograms of oriented gradient (HOG), grey level co-occurrence matrices (GLCM), Gabor filter processing, and image feature vectors (IMG-FV). The IMG-FV is a feature of the image itself. The different features are then combined to obtain the feature descriptor of images. 3) Unsupervised anomaly detection methods are used to classify the combined feature data. The above steps are described in further detail as follows: All images are resized to 224 × 224 pixels, followed by feature extraction. All images are then reshaped to obtain the image data. The size of the image data is 56 × 896. Principal component analysis (PCA) is then used to reduce the dimensions of the image data and HOG features due to

1) FEATURE EXTRACTION
As aforementioned, four types of texture features are extracted in our proposed method. These features are widely used in face recognition, medical image processing, remote sensing classification and industrial fault detection [17]- [20]. A description of each type of feature is as follows.
A local binary pattern is a type of visual descriptor used to describe local textural features of images. The improved LBP operator [21] has the advantages of rotation invariance and gray invariance, and can adapt to textural features of different scales.
The grey level co-occurrence matrix is calculated from a gray-scale image. It is used to describe texture by studying the spatial correlation characteristics with respect to interpixel    distance (δ) and orientation (θ) [22]. A total of six statistics including contrast, dissimilarity, homogeneity, correlation, entropy and angular second moment (ASM) are applied to co-occurrence probabilities to generate the textural features.
A Gabor filter is a linear filter used for textural analysis. Its frequency and direction expressions are similar to those of a human visual system. It is not sensitive to light changes and provides scale and orientation parameters [23].  A histogram of oriented gradient is a feature descriptor used in computer vision and image processing for the purpose of object detection [24]. It convolves a filter kernel with an input image to get two-dimensional gradients of the image.

2) ANOMALY DETECTION
Four types of anomaly detection algorithms are used for fault detection: Isolation forest (iForest), Gaussian distribution (Gaussian-D) based anomaly detection, one-class SVM (OC-SVM) and local outlier factor (LOF). The iForest is a fast anomaly detection method based on ensembles. It has a linear time complexity and high precision [25]. It uses a random hyperplane to cut the feature data space until there is only one data point in each subspace, which is then used to build a decision tree. The average value of the feature data in each tree is used to define a threshold or a boundary value. The data points below this threshold are considered as abnormal.
The Gaussian-D based anomaly detection algorithm is widely used in many scenarios [26]- [28]. The algorithm is based on the core assumption that the inlier, i.e., the normal data, is Gaussian distributed. We estimate the inlier location and covariance in a robust way using the Fast-MCD algorithm without additional processing [29]. The dataset is transformed into an n-dimensional Gaussian distribution dataset. Its probability density function is determined, and a threshold is calculated. Based on Gaussian probability (p) of a data point and threshold ε, a point with p < ε is considered an outlier, while p > ε identifies an inlier.
The SVM is one of the most successful machine learning techniques typically associated with supervised learning. There are extensions of SVM such as the OC-SVM method that can be used to identify anomalies as an unsupervised problem. The method basically separates all the feature data points from the origin by a hyperplane and maximizes the distance of this hyperplane to the origin [30], [31], thus minimizing the influence of the outliers.
The LOF is based on the concept of local density. A typical distance at which a point can be ''reached'' from its neighbors is applied to estimate the local density [32]. The local deviation of a given data point with respect to its neighbors is used to find outliers.

B. DATA ACQUISITION AND PRE-PROCESSING 1) INSPECTION EQUIPMENT
We use a new non-powered pipeline inspection equipment designed by us specifically for this application [33], [34]. Its schematic diagram and photo are shown in Fig. 2. It consists of a robot equipped with a high-resolution fisheye camera that can provide 360-degree panoramic and 220-degree wideangle video surveillance data. It can move continuously, and record video data of the sewer pipeline walls, which make it appropriate for precise inspection of sewer pipeline faults. The equipment floats at speeds of between 0.3 m/s and 0.8 m/s, approximately, depending on the water stream speed.
The equipment is connected to a mobile phone or tablet application and controlled through WiFi. The unpowered design is used to move inside the pipeline along the flow of VOLUME 8, 2020 water. Due to its weight and design, the equipment is stable and can acquire data continuously in the vertical direction. The inspection procedure using the equipment inside the sewer pipeline is shown in Fig. 3.

2) DATASET AND EVALUATION
Underground sewer pipeline video data were collected in the city of Shenzhen. We converted the video data into a sequence of images and extracted different features from the sequence to generate sequence signal features. The anomaly detection algorithm was then used for fault detection.
To study the influence of different features on fault detection accuracy, we tested different combinations of features and the image feature vector to generate a total of seven different feature groups. A description of the seven groups is as follows: 1) The HOG, GLCM, Gabor and LBP features and the image feature vector are combined to generate an ''All-Feature'' group.
2) The HOG, GLCM, Gabor and LBP features are combined to give a ''Four-Feature'' group.
In order to verify and analyze the efficacy of each feature group, we built two types of datasets: large-scale datasets and single-segment small datasets.
The first type (Dataset-1) has 8952 images in total extracted from multiple sewer pipeline videos. It contains 1514 images with faults. The images were acquired in a variety of underground network environments, such as PVC and concrete material sewer pipelines. A few samples of the dataset are shown in Fig. 4.
The second type of dataset (Dataset-2) includes two videos, Video-1 and 2, which were recorded between two manholes. Video-1 consists of 899 images. Out of these images, there are 194 images that include faults. A few samples of this dataset are shown in Fig. 5. The dataset of Video-2 consists of 1260 images, out of which, 128 images include faults. A few samples of the dataset are shown in Fig. 6.
The resolution of the original video is 1920 × 1080. Due to the original video characteristics, which are shown in Fig. 7, the frame needs to be cropped to obtain valid image data. Currently, the size of valid region of video has slight differences due to recording by different equipment. The resolution of our dataset includes frames of size 1100 × 1080 and 950 × 700 after cropping the frame. The aspect ratio of the valid region of the frame is approximately one, and we found that a resolution of 224 × 224 is better than 512 × 512 or 112 × 112 or any smaller or larger resolution for our experiment. Therefore, all cropped frames were resized to 224 × 224 before feature extraction and the reshaping step. These two types of datasets cover a variety of sewer pipeline environments with different types of faults including breaks, cracks and deformation. In order to evaluate different feature combinations and find the best solution, we apply four different anomaly detection algorithms to these datasets.
The overall accuracy, precision and recall are well-known measurements that can be used to assess the performance of a classifier. Five different parameter indicators are used to evaluate the fault detection algorithms, and their definitions are given in Eqs. (1)(2)(3)(4)(5). Suppose we have P positive samples, i.e., fault-free samples, and N negative samples, i.e., fault samples. Our confusion matrix is shown in Table 1 where TP, TN, FP and FN indicate true positive, true negative, false positive and false negative, respectively, and N = TN + FP, P = TP + FN . The precision and recall given by the true negative rate and negative predictive values reflect the accuracy of the fault detection.

IV. RESULTS AND DISCUSSION
The four classification algorithms described in Section III-A are applied to each of the feature groups. All the evaluation results for Dataset-1 are shown in Table 2. Four of the five top-performing combinations consist of the Gaussian-D algorithm combined with other feature groups. These combinations achieve the best results with an overall accuracy of 88.0%. The fifth combination shown in Table 2 corresponds to the iForest algorithm used with Gabor features. In order to better evaluate the classification performance, the ROC curve of each anomaly detection algorithm applied to each feature group is shown in Fig. 8. It can be seen that the Gaussian-D based anomaly detection algorithm has the best performance, followed by the iForest algorithm.
For Video-1, all the evaluation results are shown in Table 3. The iForest algorithm combined with GLCM features has the highest overall accuracy of 90.2%. In order to better evaluate the classification performance, the ROC curve of each anomaly detection algorithm applied to each feature group is shown in Fig. 9. The classification performance of OC-SVM and LOF algorithms is poor with all of the feature groups. The Gaussian-D and iForest algorithms show better performance.
For Video-2, all the evaluation results are shown in Table 4. The best result is obtained with the combination of the Gaussian-D algorithm and the Gabor features. The ROC curve of each anomaly detection algorithm applied to each feature group is shown in Fig. 10. The iForest algorithm combined with the Gabor and GLCM features also shows good performance, achieving an overall accuracy of approximately 90%. The OC-SVM and LOF show poor performance.
Figs. 8-10 can be compared to review the results with different datasets and ROC curves for different algorithms. It can be observed that the performance of OC-SVM and LOF algorithms is inferior to that of the Gaussian-D and iForest algorithms. Although no single combination performs the best across all of the different datasets, the Gaussian-D and iForest algorithms show good performance in most cases. We observe that the Gaussian-D algorithm has the best performance, irrespective of the type of feature combination used. The iForest algorithm is the second best in terms of performance. It can also be observed from Tables 2-4 that either the GLCM or the Gabor features result in the best performance, and there is no single feature group that always outperforms the other feature groups. The variation in performance with different features is related to the environment of each pipeline section. Different pipeline materials and states of the pipeline discharge can lead to varying fault types. For different datasets, the top five algorithm and feature combinations consist of the Gaussian-D or iForest algorithm combined with other feature groups. The feature groups including GLCMs, GABOR, and IMG-FV show better results, which emphasizes their effectiveness in extracting features in the sewer pipeline environment. In real pipeline scenarios, anomaly detection algorithms combined with texture feature extraction can be used for fault detection from sewer pipeline videos. This method can also be extended to other video fault detection scenarios.
Based on these experimental results, we can rank the algorithm and feature performance for sewer pipeline fault detection. In terms of the algorithms, the Gaussian-D and iForest algorithms perform almost the same as each other, and the OC-SVM and LOF algorithms perform almost the same as each other. The former group is ranked higher than the latter group. In terms of the features, the Gabor and GLCM features provide almost identical performance. These features are ranked higher than the original Image data features, which are ranked better than the HOG and LBP features.
As the results show, both GLCM and Gabor features are always included in the five best-performing feature groups, which means that these two features can provide reasonably good results for all datasets. As GLCM and Gabor features can better reflect the textural changes between the fault and fault-free images, the abnormality detection algorithms can capture this change effectively.
Each anomaly detection algorithm and feature has its own advantages in different scenarios. However, our processing results show that the proposed fault detection method is effective in sewer pipeline environments. The method can detect outliers (faults) from unknown data without using a trained classifier. As it is difficult to know the shapes of different types of sewer faults in advance, our proposed method is more in line with real scenarios. The OC-SVM algorithm is known to be sensitive to outliers and thus does not perform very well for outlier detection. The Gaussian-D algorithm assumes the data to have a Gaussian distribution, and shows more robust performance in the experimental results. The iForest algorithm detects anomalies (faults) based on the concept of isolation without employing any distance or density measure. It shows better performance than the LOF algorithm. LOF has a more local focus but produces more errors when the data is noisy. Out of all these algorithms, the Gaussian-D and VOLUME 8, 2020 iForest algorithms outperform the other algorithms. In summary, the Gaussian-D or iForest algorithms can be used as effective auxiliary methods, and avoid the problem faced by deep learning methods that require training on annotated data sets.

V. CONCLUSION
This paper applied anomaly detection methods for underground sewer pipeline fault detection using video data. The data was collected by a new type of sewer pipeline inspection equipment. We carried out a comprehensive evaluation of various anomaly detection algorithms and feature groups. The evaluation showed that the Gaussian-D and iForest algorithms with feature groups including GLCM or Gabor can achieve good results. Anomaly detection algorithms can reduce the workload required for data annotation or manual identification. In the absence of any labeled data, these algorithms are more practical, and can improve the efficiency of sewer pipeline fault detection and reduce costs associated with establishing a database of faults. As the proposed method is based on unsupervised machine learning, it can reduce the cost of manual detection or data annotation, which is required for deep learning-based methods. Deep learning-based methods have been used to detect faults in sewer pipelines based on CCTV videos or images in several recent research studies, such as Faster-RCNN, YOLO and SSD [15] and have shown effective performance. However, since these methods are supervised learning, they require a large annotated dataset for training. In contrast, the anomaly detection algorithm is an unsupervised learning method so it is difficult to perform quantitative comparison. In future work, we shall consider semi-supervised learning methods to achieve more detailed classification of fault types.