A Machine Vision System Based on Driving Recorder for Automatic Inspection of Rail Curvature

Because of long distance of railway lines, it is difficult to find an appropriate method to inspect the rail track condition efficiently and accurately. In this article, a machine vision system based on driving recorder and image signal processing is proposed to evaluate the rail curvature automatically. The proposed machine vision system consists of four modules including the video acquisition module, the image extraction module, the image processing module, and the track condition assessment module. Three classic edge detection methods are adopted and compared for rail edge detection. In line with the videos of driving recorder, coordinate systems for train and rail are defined in the Lagrangian space, and the track curvature is estimated using the proposed chord offset method and double measurement method. For evaluating the track condition, an index describing the concordance between the train and track is defined. In the case study, a set of videos from the driving recorders of trains during their in-service operations are analyzed by the proposed technique, and the obtained results are verified by comparison with those obtained by a track geometry inspection vehicle. It is shown that the proposed technique can evaluate the track curvature accurately. Moreover, the influence of the position of deployed driving recorder, the focal length and anti-shake of camera on the accuracy of evaluation results is discussed. It is testified that the proposed technique provides a simple and reliable way to inspect the track curvature.

of human lives. According to the database from the federal railroad administration of USA, 18% train accidents were due to the track geometry and 16% caused by the defects in track components [1]. For the safe operation of railway system, track inspection is required to be performed regularly in order to maintain the healthy condition of track [2]. Track inspection generally includes a wide variety of specific tasks, ranging from locating and evaluating the condition of different track components to monitoring rail surfaces, alignments and curvatures [3]. According to the report of Network Rail, track geometry variation exists particularly over curved track, which aggravates the likelihood of rail contact fatigue clusters [4]. In addition, the track geometry degradation can affect negatively track performance and safety [5]. Hence, track geometry is a critical issue of track inspection to maintain track functionality and should be inspected at predetermined times to ensure an acceptable condition of the track geometry.
Track inspections were often conducted visually by railroad inspectors. However, the manual inspection is very errorprone and time-consuming for railway managers/engineers to perform those tasks, especially for long-term and large-scale deployment [3]. Thus, it is essential to develop scalable, quick, and cost-effective track inspection systems. Rail inspection This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ vehicles equipped with different monitoring units and sensors have been adopted for track geometry condition detection with the purpose of maintenance planning and track classification [6], [7]. However, rail inspection vehicle is expensive and still needs operators [4]. Recently, a variety of online and/or onboard sensing techniques using sensors such as strain gauges [8], fiber optic sensors [9]- [11], bogie-mounted accelerometers [12], car-body sensory devices [13], and acoustic emission [14]- [16], have been increasingly used to inspect rail geometry and/or track condition. However, these techniques sometimes fail to satisfy performance requirements including stability, accuracy and durability under specific conditions [17]. Novel non-destructive evaluation techniques such as vision inspection [18]- [20], Doppler LIDAR technology [4], and laser dynamic triangulation [21]- [25] have emerged as well. Quite a few machine-vision based automatic systems [26]- [32] have been proposed to cope with various tasks including detecting rail irregularities, rail surface, rail anchors, turnout components, crib ballast, and defects in fasteners. The vast majority of these systems focus on the inspection of specific rail components by using an array of cameras or laser equipment [2].
In this article, an alternative image identification system is proposed that makes use of the driving recorders equipped on in-service trains to evaluate track geometry and condition. By using this technique, the geometry of a track along its whole length can be 'scanned' quickly and frequently to prevent potential track-induced train accidents and reduce the track maintenance cost. The remainder of this article is organized as follows. Section II presents the proposed track curvature and condition detection system. Section III describes three classic edge detection methods for rail edge detection. The chord offset method and double measurement method for track curvature calculation are presented in Sections IV and V, respectively. Case studies, results, and discussions are provided in Section VI, followed by our conclusions in Section VII.

II. SYSTEM OVERVIEW
The proposed track curvature and condition detection system includes four functional modules: the video acquisition module, the image extraction module, the image processing module, and the rail condition assessment module, as shown in Fig. 1. The driving recorder, computer and image processing software constitute the machine vision part of the inspection system, which mainly accomplish the function of track image data acquisition and analysis. The functionality of each module is detailed as follows.
1) The video acquisition module: The videos are collected from the driving recorder equipped on each train during its routine operation. The camera with 1080 pixels of resolution is in general used for driving recorders in which the capture speed is 30 fps. The mounted position of the driving recorder should be on the front glass of locomotor to catch the full view of track; 2) The image extraction module: The videos are converted to images according to the frames of the videos. The extracted images will be used in the image processing module through the developed image processing software;  3) The image processing module: The image processing consists of the setting of coordinate system, detection of rail edge, and feature detection of track geometry. The edge detection techniques will be introduced in Section III and the methods for the setting of coordinate system and detecting track geometry will be detailed in Sections IV and V; 4) The track condition assessment module: A track condition assessment index will be defined in this module to evaluate the concordance between the train and track, which will be specified in Section V.
The developed system provides an alternative way to inspect the track curvature and condition using videos from the driving recorder of a traveling train. While the proposed system has the potential for applications in track geometry inspection as well as in track component detection, this article will focus on the procedure of track geometry inspection such as track curvature estimation using the proposed chord offset method and double measurement method.

III. EDGE DETECTION
The edge identification of track is an essential part to calculate the location of track using machine vision technology. As shown in Fig. 2, the edge of rail has special characteristics of smooth surface and color consistency due to the prolonging friction between the wheels and rail. These features of rail edge are conducive to edge detection of rail in image processing. The primary process of edge detection is to detect abrupt changes in image intensity which can be accomplished by use of the first or second order derivatives. The diagram of the edge detection procedures is shown in Fig. 3.
A filter is designed to approximate the first derivative. The magnitude and direction of intensity changes of an image G  are calculated by the gradient operator, defined as [33] The magnitude U and direction α of the gradient operator at an arbitrary location are calculated as To obtain the gradient, numerical approximations of partial derivatives ∂G/∂ x and ∂G/∂y are calculated at every pixel of the image. The Prewitt algorithm and the Sobel algorithm are two common methods to perform this calculation [33].
In the Prewitt method [34], these approximations are obtained by filtering the image with the two masks of size 3 × 3 shown in Fig. 4(a). The difference between the third and first rows approximates the derivative in the x-direction, and the difference between the third and first columns approximates the derivative in the y-direction. The Sobel algorithm [35] uses more weights on the central coefficients of the masks (shown in Fig. 4(b)), thus providing better image smoothing. Applying the respective operators, the gradient is obtained using (1). Finally, edges in the image are determined by thresholding the gradient magnitude. The above two edge detection methods do not consider the noise in the image. The Marr-Hildreth algorithm [36] takes a step forward to reduce noise before detecting edges by finding zero crossing points of the second derivative of the image. Convolution of the image G(x, y) with a Laplacian of Gaussian (LoG) kernel is defined as Then zero crossings of the image can be found to determine the location of edges once the second derivative is obtained. The results of rail edge detection using these three algorithms are shown in Fig. 5. It is observed that the results using the Prewitt and Sobel algorithms are not good enough to detect both sides of rail edge. The rail edges inside the red circles in Figs. 5(a) and 5(b) fail to be detected; whereas, the two edges of each rail are detected unambiguously in Fig. 5(c) using the Marr-Hildreth algorithm. The algorithms using the first derivative (Prewitt and Sobel) implement easily and run extremely fast because their main manipulation is a convolution with a very small kernel (3 × 3 pixels) [33]. The algorithm using the second derivative (Marr-Hildreth) is more accurate involving more manipulations, since convolutions with larger kernels are performed.
IV. CHORD OFFSET METHOD Track curvature measurement is an essential part of evaluating track geometry and maintaining rail functionality [4]. The mid-chord method [37] is one of the common techniques to measure track curvature. In conformance with the definition of curvature, this method utilizes a chord of a certain length subtending the degree of curve. Then the versine or length from the center of the chord to the rail is measured to determine the radius of curvature using the Pythagorean theorem. In the railway measurement, a 62-foot chord is usually used to make a mid-chord measurement in that each inch from the midpoint (31 feet) to the rail is equal to one degree of curvature [4]. Even though this method is simple to implement in track geometry testing, it is not easy to realize in the image processing. In view of this, a chord offset method is proposed in this study to calculate the track curvature in compliance with vision-based detection.

A. Coordinate System
In the image processing, the primary step is the setting of coordinate system. As shown in Fig. 6, for most train vehicles, the vehicle body is seated on bogies and the two bogies of each vehicle form the end points of a chord placed on curved track. The vehicle body represents an extension of this chord. The moving orientation of the train is along the chord direction of the curved track. Considering these features of railway, we employ the chord of the curve as the benchmark of the rail coordinates in the image processing. As shown in Fig. 7, R is the curvature radius of the curve path and the two ends of the chord represent the positions of the front and rear bogies beneath a coach. The direction of chord (y-axis) is the travel direction of train and the normal direction (x-axis) is set on the rail plane and perpendicular to the travel direction of train. The direction z is perpendicular to the plane formed by the chord and its normal line. The established Lagrangian coordinate system is related to the track. The coordinate system for the vehicle is the same as the track system when there is no relative motion between the vehicle and track. The coordinate systems for the vehicle and track when there exists a relative motion between the vehicle and track will be discussed in Section V.

B. Calculation of Track Curvature
The chord offset method is then applied to calculate the track curvature in the established coordinate system. As shown in Fig. 7, B 1 and B 2 are the locations of two bogies on the ends of chord, which in the coordinate system are (0, −y b ) and (0, y b ), respectively. The measurement point M is at the curve line whose location is (x m , y m ). The mid-ordinate offset is assumed to be c. According to the Pythagorean theorem, the following equations can be obtained: The mid-ordinate offset c is not easy to obtain in the image processing. To eliminate the mid-ordinate offset in the above equations, (4) is subtracted from (5), from which we obtain: By substituting (6) into (5), the curvature can be expressed as: where κ is the curvature. According to (7), the curvature of the curve path can be calculated once the coordinate values of the measurement point and the spacing of bogies (chord length) 2y b on the curve path are determined. It is worth mentioning that the points B 1 , B 2 , M are still in the same curve when the two bogies beneath the locomotive are within a transition zone of rail. At the short time when the two bogies are respectively at the straight and curved lines crossing the transition zone, the estimated curvature is the average curvature of the two lines.
As the offset distance is usually much smaller than the chord length of a curved railway line, the value of |x m /y m | is far less than 1. In this case, (7) can be simplified as: By this way, the curvature of a curve line is determined by using (7) or (8) when relative motion does not occur between the vehicle and track. When there exists a relative motion between the vehicle and track, the coordinate systems for the vehicle and track are no longer the same. In the latter situation, the method for calculating the curvature is different and will be provided in the next section.  V. DOUBLE MEASUREMENT METHOD Ideally, the conditions of track and train are good enough to eliminate the relative motion between the track and train. In such situation, the track curvature can be estimated by using (7) or (8) in the image processing. Considering the complex operation environment of a train moving on track, relative motion between the train and track is often inevitable. In this situation, a double measurement method is proposed in this study to calculate the track curvature.

A. Double Measurement
The coordinate systems for the vehicle and track will be different once relative motion between the train and track occurs. The relative motion between the train and track induces the deriving recorder to take videos from a different vision direction. As shown in Fig. 8, the blue triangle plane represents the view plane when the coordinate systems for the vehicle and track are coincident, and the view plane becomes yellow triangle plane when there is relative motion between the train and track. This means that the measured track curvature in the image processing will be contingent on the deployed location of the video. The curve measured by videos from different visual direction is called visual curve. As illustrated in Fig. 9, there are two visual curves from two different visual directions. The observed points M and M on the two visual curves correspond to the same point on the track but are collected from different visual directions. The curvature of visual curve calculated by (7) or (8) is named visual curvature. Obviously, the visual curvature is not the true curvature when relative motion between the track and train exists. To obtain the real curvature, the double measurement method is proposed here to eliminate the observation error. As shown in Fig. 9, two points (M and N) on the curve are considered. In accordance with the Pythagorean theorem, the following equations can be obtained: From (9), we can obtain: By substituting (12) into (10), R 2 − y 2 b can be expressed as: and then the curvature can be derived by substituting (13) into (11) as: In (14), the curvature is determined by two points on the curve line without the axle spacing (chord length) on the curve path. Akin to (7), (14) can be simplified as Both (8) and (15) can be utilized to estimate the track curvature in practice. However, (15) is a more general formula to estimate track curvature in consideration of the relative motion between train and track. The double measurement method provides an effective approach to evaluating track geometry in the image processing.

B. Concordance Between Train and Track
As aforementioned, the track curvature evaluated by (15) takes into account the relative motion between the track and train. To some extent, the difference between the results obtained by (8) and (15) reflects the relative motion between the train and track. If concordance between the train and track is good enough, there will not be relative motion between the track and train. So, the concordance between the train and track is defined in this study to quantify relative motion between the train and track. The formula of evaluating the concordance between the train and rail can be expressed as where T is the evaluation period; W is the standard curvature; κ 1 and κ 2 are the curvatures calculated by (8) and (15), respectively. The larger the index A, the poorer the concordance between the train and track. The index A can be used to evaluate the condition of track, which will be discussed in Section VI.

VI. RESULTS AND DISCUSSIONS
To verify the effectiveness of the proposed track curvature and condition detection system, three sets of videos taken from the driving recorders of trains of the same type traveling on a rail route during different time periods are first analyzed, and the results are compared with those obtained by track geometry inspection vehicle. Then, another three sets of videos from the driving recorders of three trains of different types traveling on the same railway are used to evaluate the track condition by the proposed technique. Finally, the influence of focal length and anti-shake on the accuracy of results is studied.

A. Verification Example
Three  Fig. 10. It is seen that the track curvature obtained by the proposed technique using the videos taken on 20 November 2018 highly agrees with that obtained by the track geometry inspection vehicle ( Fig. 10 (a)), verifying the effectiveness of the proposed image processing method in evaluating the track geometry. The root mean squared error (RMSE) between the two curvatures is 1.66×10 −4 1/m. The results obtained by the videos taken on 21 February 2018 and 21 October 2018 are illustrated in Figs. 10 (b) and (c). The track curvature in the eight months before the derailment accident was reasonable, but an anomaly in the track curvature occurred about three seconds before the derailment accident (an abrupt change in curvature since the 28th second in Fig. 10(c)). The results indicate that the proposed technique has the potential to be pursued for pre-warning of derailment.
As mentioned in Section V, the vision direction can significantly affect the evaluation results. In this regard, the double measurement method has been proposed to account for the effect of vision direction. For demonstrating the efficacy of this method, four cameras of the same model were mounted on a train but at different positions to record the track simultaneously. The railway line is about 11 km in length. The vision directions from the four cameras and the track curvatures evaluated by the double measurement method using different videos are shown in Fig. 11. It is seen that the track curvatures evaluated by this method using the videos from different vision

B. Evaluation of Track Condition
Three sets of videos from the driving recorders of three trains of different types (Taroko, PuYuma, and Revival Expresses) are adopted here to evaluate the track condition of a railway line by the proposed technique. The railway line is about 9 km in length and three kinds of trains travel on it every day. Among the three expresses, the Taroko and PuYuma Expresses are tilting trains. Three videos from the driving recorders on the Taroko, PuYuma, and Revival Expresses were taken on 20 April 2017, 7 August 2017, and 22 February 2018, respectively. In evaluating the concordance between the train and track using index A defined in (16), the interval time T is  taken as 3s and the standard curvature is the one when the mid-ordinate offset is 7 cm.
In Fig. 12, three pictures are extracted from the automatic image processing using the videos of the Taroko, PuYuma, and Revival Expresses, respectively. The establsihed coordinate plane and the calcualted curvature are also shown in Fig. 12. In light of these, the track curvatures and concordance between the train and track along the railway line are estimated and plotted in Fig. 13. The curvatures of each point on the railway line obtained by (8) and (15) are represented by red and blue lines respectively in the figure; and the green line denotes the concordance index A of the train and track. It is seen that both the track curvature and the concordance between train and track resulting from the three videos are in good agreement, since the three trains were traveling on the same railway line. The RMSE between the curvatures in Figs. 13(a) and (b) is 6.73 × 10 −4 1/m, while the RMSE between the curvatures in Figs. 13(a) and (c) is 3.32 × 10 −4 1/m. It is also observed that the index A reflecting the concordance between train and rail is higher for the PuYuma Express than for the Taroko and Revival Expresses. For the track sections in the yellow block in Fig. 13, the curvatures evaluated by (8) and by (15) are apparently different where the index A becomes abnormally large. This indicates that the track condition is poor in this segment compared with the others.

C. Influence of Focal Length
The focal length of camera influences the angle of view. The shorter the focal length, the wider the angle of view and the greater the area captured. The longer the focal length, the smaller the angle and the larger the object appears to be. When we calculate the track curvature using image, the image with larger track targeted and enough area captured is preferable. To figure out the influence of the focal length of camera on the evaluation results, a test of two cameras with different focal lengths installed on a train was performed. As shown in Fig. 14, the focal lengths of the two cameras are 2.66 mm and 4.2 mm, respectively. The two cameras are installed at the same visual position to record the condition of track synchronously. The testing track is about 8.6 km and the total number of analyzed images from each camera is 9940 fps. The images from the two cameras, the establsihed coordinate plane, and the estimated curvature are shown in Fig. 15. It looks that both images can capture enough track area. However, the track object targeted by the image from the camera with focal length f = 4.2 mm is larger than that from the camera with focal length f = 2.66 mm. The track curvatures estimated using the two videos are plotted in Fig. 16. It is seen that the curvature evaluated using the video taken by the camera with focal length of 2.66 mm (blue line) is in close agreement with that by the camera with focal length of 4.2 mm (red line). A close-up view of the two curvatures is given in the lower panel in Fig. 16. Only very tiny difference exists, which implies that the focal length has little influence  on the evaluation results. The results from an image targeting larger track may provide a better accuracy.

D. Influence of Anti-Shake
Most digital cameras have the function of anti-shake which filters the noise frequency higher than 0.5 Hz. However, the vibration frequencies of train are about 1.0 to 3.0 Hz. This means that the evaluation results using the images taken by a camera with anti-shake activated may be less accurate due to eliminating the vibration of train. To verify this, a test with two cameras with and without anti-shake installed on a train was conducted. The track curvatures estimated using the videos collected by the two cameras respectively are plotted in Fig. 17. A noticeable difference between the two estimated curvatures is observed and the curvature from the videos taken by the camera with anti-shake (red line) is smoother than the other one (blue line). As a result, the anti-shake function in camera is likely to trim down the accuracy of evaluation results.

VII. CONCLUSION
A machine vision-based rail track condition assessment system using driving recorder was demonstrated to be capable of estimating track curvature and evaluating track condition. Three edge detection methods were applied to detect the edges of rail. The chord offset method and the double measurement method were proposed to calculate the track curvature in image processing, while an index reflecting the concordance between the train and rail was proposed to evaluate the track condition. The proposed track curvature estimation technique was verified by comparison with the results obtained by a track geometry inspection vehicle. Unique images taken by driving recorders equipped on in-service trains traveling on a rail route before and during a derailment accident as well as after repairing the track were used to examine the feasibility of the proposed technique for rail condition assessment.
Field tests were also conducted to verify the accuracy of the proposed chord offset and the double measurement methods for track curvature estimation and the influence of various factors on the evaluation results. It is shown that the evaluation results are not affected by the position of the deployed driving recorder when the proposed double measurement method is adopted. The higher focal length is preferable under enough capture area to guarantee the accuracy of results. It is observed that the anti-shake in camera can negatively affect the evaluation results. This study concludes that the proposed technique using driving recorders equipped on trains provides a viable means for track curvature and condition evaluation.