Abnormal road surface detection using wheel sensor data

In this manuscript we investigate methods for abnormal road surface detection using 3D force sensors implanted into the wheels of a vehicle. This research is realized as a collaboration between the Hungarian Institute of Technical Physics and Materials Science (MFA), Institute for Computer Science and Control (SZTAKI) and the department of Numerical Analysis of Eötvös Lóránd University (ELTE). The sensor in question was created by researchers at MFA. The hardware’s technical description will not be discussed here in detail, rather we focus on the sensor’s applications and the signal processing methods needed to implement them. Nevertheless we provide a basic overview of the sensor and the measurements that it produces below. A single sensor has four output bridges, which measure a change in resistance when the sensor is subjected to some outside force. A simple (albeit not perfect) relationship between the forces acting on the sensor and the produced outputs is described by


Introduction
In this manuscript we investigate methods for abnormal road surface detection using 3D force sensors implanted into the wheels of a vehicle.This research is realized as a collaboration between the Hungarian Institute of Technical Physics and Materials Science (MFA), Institute for Computer Science and Control (SZ-TAKI) and the department of Numerical Analysis of Eötvös Lóránd University (ELTE).
The sensor in question was created by researchers at MFA.The hardware's technical description will not be discussed here in detail, rather we focus on the sensor's applications and the signal processing methods needed to implement them.Nevertheless we provide a basic overview of the sensor and the measurements that it produces below.
A single sensor has four output bridges, which measure a change in resistance when the sensor is subjected to some outside force.A simple (albeit not perfect) relationship between the forces acting on the sensor and the produced outputs is described by where F x , F y and F z are the different forces acting on the sensor and the ∆V directions denote the measured changes in resistance at each bridge.It is important to emphasize, that the forces in (1) refer to the forces acting on the sensor itself, the relationship between the forces acting on the tyres of the vehicle and the sensor's output is much more complicated.
The wheel sensors are located on the inner wall of the two front tires of the test vehicle (one sensor on each side).Because of this, the output signals are produced in response to tyre deformation.When the vehicle is in motion, the forces acting on a single point of the tyre (and causing deformation) depend on the wheel angle, thus the produced output signals are quasi-periodic.Figure 1 illustrates the signals generated at each bridge of the sensor.
A single period corresponds to a full rotation of the tyre.Since the sensor is most-excited when it is close to the ground, the resulting periods produce quasi-compact signals.The quasi-periodic and quasi-compact behavior of the wheel sensor signals make them very similar to many biological signals such as ECG.This similarity will be later exploited in the introduced signal processing algorithms.
The discussed sensor is well suited for the detection of road surface abnormalities, because of the direct relationship between tyre deformation and the output signals.The detection and subsequent classification of road abnormalities is a well researched [3] and important topic.Gathering and sharing information about road quality can decrease maintenance costs and several recent information sharing frameworks [2, 1, 3] have been proposed for this purpose.
In this report, we show through experiments that the investigated wheel sensors are indeed well suited to detect road surface abnormalities.We propose, discuss in detail and compare several abnormality detecting algorithms.
The rest of this report is organized as follows.In section 2.1 we discuss the preprocessing steps applied to the output signals of the wheel sensor.We then describe naive approaches with well-known classification schemes in 2.2.The main findings of this manuscript can be found in sections 2.3 to 2.5.We discuss an appropriate way to model the output of the wheel sensor using socalled adaptive Hermite-functions [8] in 2.3.Through an experiment, we then demonstrate how adaptive Hermite-functions can be applied for road surface abnormality detection in 2.4.Finally, in section 2.5, we combine the previous findings into a robust classification scheme utilizing a state-of-the-art neural network architecture called VP-NET [7].The conducted tests, their results and subsequent discussion can be found in sections 3 and 4.

Road Surface Abnormality Detection with Wheel
Sensor Data In the following sections we detail several approaches for road abnormality detection based solely on wheel sensor measurements.

Preprocessing and data description
In the first section of this report we observed the quasi compact and quasi periodic nature of the wheel sensor data.In order to use wheel sensor based signals for surface abnormality detection, we analyze the properties of each full period (corresponding to a full rotation of the tyre), thus we need to segment the measurements.The segmentation algorithm used to produce the below results is based on the ECG-segmentation method described in [9], however new steps and various new parameters had to be introduced to adapt the method to wheel sensor data segmentation.The specifics of the segmentation algorithm will not be detailed here, as currently the test vehicle is being equipped with accurate wheel-angle measuring sensors, making any subsequent segmentation obsolete.
Once the data from the different bridges of the wheel sensor (see Figure 1) has been segmented, we can label each period as "normal" or "abnormal" using our ground truth data.That is, if some acceleration data which has been detected as caused by surface abnormality occurred during the current period, the entire period is labeled "abnormal".Ground truth generation is described in section 3.
Because of the imperfect nature of the segmentation algorithm and noise present in the measurements, the first and last points of a period of wheel sensor data may not be equal.In order to preserve the quasi-compact property of each period, we subtract the line connecting the first and last values of the period.
The number of data points which make up a period changes with the vehicle speed.For the easier handling and storage of the data, zero padding is applied to each segmented period.The maximal length of a period is identified as 500 data points, any periods longer than this (for example if the vehicle stood still for sometime) are disregarded.Periods of fewer than 500 data points, are zero padded to match this length.An example period after the above preprocessing steps labeled "normal" and another one labeled "abnormal" are given in Figure 2.These preprocessed periods will be referred to as "samples" henceforth.The samples are then standardized and their order is randomized.Finally, the samples are split into training and test sets.The properties of these sets are discussed in section 3.

Road abnormality detection with well-known classifiers
Once the wheel sensor signals have been preprocessed, we can utilize classification schemes to identify the samples corresponding to abnormal road conditions.In order to provide a benchmark result for more sophisticated classification approaches, an SVM classifier using Gauss-kernel was trained and tested with the samples.
Well known classifiers such as fully connected neural networks and convolutional neural networks were also implemented and used to classify the wheel sensor data.The networks considered here all used the binary cross-entropy loss function and were trained using the powerful ADAM optimizer [10].The exact number of layers and neurons per layer was determined through a grid search of the hyper-parameter space.The best performing network architectures are detailed for each case in section 3.

Modeling wheel sensor data using adaptive Hermitefunctions
The main assumption for wheel sensor based road abnormality detection is that samples corresponding to abnormal road conditions will contain more noise than those measured on a normal surface.Comparing noise levels on the preprocessed samples however is problematic, because (especially at high velocity) the signals from the wheel sensor contain high frequency components.One way to measure noise levels would be to model the samples using smooth functions, subtract the approximation from the measured sample and check the noise levels of the residual.Below we briefly describe adaptive Hermite-functions as introduced in [4] and provide insight into why this function system is especially well-suited to model wheel sensor data.
Let us denote the m-th Hermite-polynomial by h m (x), (m ∈ N).These polynomials are orthogonal on the weighted Lebesgue-space L 2,w (R), where ( Using the Hermite-polynomials we can acquire the so-called Hermite-functions These functions provide a (complete) orthonormal function system on L 2 (R) and so Next we detail some properties of the Hermite-functions, which make them particularly suitable for the approximation of quasi-compact signals.The first few Hermite-functions are depicted on figure 3.
• Hermite-functions tend quickly to zero as the argument increases: • Hermite-functions can be calculated with a (stable) second order recursion: • The derivative of Φ m can be expressed with Φ m and Φ m−1 : Taking the affine transformations of Hermite-functions yields further complete orthonormal systems which retain the useful above properties.Specifically, consider the functions Then, the function system √ λΦ t,λ m , (m ∈ N) is also orthonormal and complete on L 2 (R), thus for the partial sums S t,λ m f holds.We refer to these functions as adaptive Hermite-functions [4] henceforth.The m-th partial sum, S t,λ m f can also be thought of as the projection of f onto the subspace spanned by the first m adaptive Hermite-functions.This subspace is fully determined by the dilation and translation parameters λ > 0 and t ∈ R. For a fixed function f ∈ L 2 (R), and a fixed dimension m ∈ N, the error functional always has a local minimum [4].In order to get the best possible approximation for a fixed m, we need to minimize (7) with respect to t and λ.
In applications the function f , (in our case a preprocessed sample, see Figure 2) is given as a vector of discreet values.We can approximate such an f ∈ R N , N ∈ N with the linear combination of the first m discretely sampled adaptive Hermite-functions by where the components of c ∈ R m are referred to as linear parameters and the k-th column (ψ t,λ k ) of Ψ t,λ ∈ R N ×m is the discretized version of the adaptive Hermite-function Φ t,λ k (x), (x ∈ R, k = 1, . . ., m).Since the approximation depends on the λ dilation and t translation parameters in a nonlinear way, we will refer to these as nonlinear parameters.For fixed nonlinear parameters, the linear parameters can be expressed as where Ψ + t,λ denotes the Moore-Penrose pseudo inverse, and thus the projection of f onto the subspace spanned by the first m discretely sampled adaptive Hermite-functions can be expressed by We note that if we choose the sampling points as the (N real) roots of Φ t,λ N (x), then the resulting sampled adaptive Hermite system will have discrete orthogonality.The difference between the signal f and its approximation P t,λ f is called the residual and can be expressed by Similarly to (7), our goal is to find the nonlinear parameters λ > 0 and t ∈ R, such that the error functional e(c, t, λ) assumes a local minimum, where the operator P ⊥ t,λ := (I − Ψ t,λ Ψ + t,λ ) projects onto to the otrhogonal complement of the subspace spanned by the columns of Ψ t,λ .The operator P ⊥ t,λ is referred to as a variable projection operator [5] and since (for fixed nonlinear parameters) the linear parameters can be expressed via the pseudoinverse, the optimization task can be simplified by min c∈R m ,t∈R,λ>0 e(c, t, λ) = min t∈R,λ>0 Such optimization problems are called separable nonlinear least squares (SNLLS) problems.In [5] it is shown that the gradient of e(c, t, λ) can be given explicitly if the partial derivatives of the adaptive Hermite-functions are known with respect to the nonlinear parameters.We note that one could use any linearly independent function system instead of adaptive Hermite-functions, however because of the reasons highlighted above these functions are suitable for modeling wheel sensor data.
We can now acquire the so-called residuals ("flattened" wheel sensor samples) with the following steps: 1. Solve (13) to determine the optimal dialation λ * and translation t * parameters for the given sample f using a gradient based optimization method, 2. Approximate f with P t * ,λ * f , (10), 3. Acquire the residual f − P t * ,λ * f , (11).
Finally we note that adaptive Hermite-functions can be further generalized by adding new parameters to the weight function w(x) := e −x 2 [8].We would like to explore approximations of wheel sensor data with these so-called weighted Hermite-systems in a future work.

Road abnormalities and the residual signal
As stated in 2.3, our main assumption regarding road abnormality detection using data from the wheel sensor was that samples corresponding to abnormal road conditions will contain more noise than those measured on a normal surface.In order to empirically verify this hypothesis, we randomly selected and examined 100 samples from each class (normal and abnormal).To estimate noise levels, we looked at the standard deviation of each sample.That is, for a sample f ∈ R N , (see figure 2), we calculated where µ is the mean of the sample f .In order to justify modeling the samples using adaptive Hermite-functions and examining the noise levels in the residuals (11), we also measured the standard deviation of the residuals acquired from the same samples.The results of the investigation can be seen on figure 4. On the figure, the standard deviation of samples recorded on normal road surfaces are colored blue, while the abnormal samples are colored red.Even though noise levels of normal samples seem to be generally less than those of abnormal samples, when the deviation is compared on the samples themselves (top of figure 4), the classes cannot be separated by a simple noise threshold.On the other hand, when noise levels were compared on the residuals (bottom of figure 4), the standard deviations of normal and abnormal cases can be easily separated using a threshold.A possible explanation as to why the noise levels of residuals can be separated more easily than the noise levels of the samples themselves follows the intuition that at high speeds, (which would occur more often on normal road surfaces), higher frequency noise-like components appear in the samples.This increases their standard deviation thus causing an overlap in the classes' noise levels.On the other hand, the residuals are created by subtracting a smooth approximation from the samples ( 11) which (if the approximation is precise enough) removes any high frequency components appearing in the signal, but has little effect on the noise.
Even though the above experiment empirically verified our main assumption on the noise levels of the different classes, comparing the standard deviation of residuals does not lead to a perfect result as can be seen on figure 4. To overcome this a more sophisticated, machine learning based approach is utilized.

Classification with VP-NET
In this section we describe the application of VP-NET to the road abnormality detection problem.VP-NET is a special neural-network architecture introduced in [7] containing so-called variable projection layers.These layers are capable of solving SNLLS problems (13) and passing the results to a conventional neural network.VP-layers have several modes of operation.They are capable of passing linear parameters, nonlinear parameters, approximations, or residuals to lower layers.
Formally, the different versions of VP-layers can be expressed by the below equations.Let f ∈ R N , N ∈ N be an input sample and m ∈ N. Furthermore let Ψ µ ∈ R N ×m , µ ∈ R p , p ∈ N be a matrix whose columns contain some discretized version of a linearly independent function system, where the system depends (in a nonlinear fashion) on the parameter vector µ.Then, a VP-layer solving the SNLLS problem and passing the optimal linear parameters can be given as These parameters can then be used to solve classification problems.This use case can be regarded as a dimension reduction and automatized feature extraction step.For regression tasks it might be more appropriate to pass the approximation (projection onto the column space of Ψ µ ) of f to the lower layers: When detecting road abnormalities, the experiments detailed in 3 show that the VP-layers performed best, when they passed on the residual signal: These results were in line with the reasoning given in 2.4.A visual representation of VP-NET is given on figure 5.

Input Hidden layers
Output layer In theory, convolutional layers equivalent to VP-layers could be constructed [7], however as can be seen in section 3, the application of VP-NET is more appropriate for the detection of road abnormalities.The reasons behind this could be traced back to several important differences between VP-NET and convolutional networks.Firstly, the optimization task is usually a lot simpler when using VP-NET.Instead of having to learn the appropriate kernel weights (which could be numerous), the VP-layer only has to optimize the nonlinear parameter vector µ.In our case for example, when the columns of Ψ µ contain discretized adaptive Hermite-functions, the parameter vector µ only contains only two components: µ := (t, λ) ∈ R 2 .By contrast the best performing convolutional network used a kernel size of 25 making the training process and the network more complicated.Another advantage of using VP-layers is that unlike the the kernel weights, the nonlinear parameters of a VP-layer can contain interpretable information.In our case for example, the dilation parameter is related to the speed of the vehicle.

Tests and Results
In order to verify the results of any wheel sensor based abnormality detection algorithm, we need to establish some ground truth data.In other words, we need to label the timestamps of the measurements which correspond to times when the vehicle encountered abnormal road conditions.This can be done by creating measurements in a controlled environment, where road surface abnormalities and the timestamps when they occur are known in advance.Although there are plans to conduct such measurements using our test vehicle in the future, in this report we relied on automatically labeled measurements.Automatizing ground truth generation can have several benefits, including access to more data for training and testing purposes.Recently many successful road surface abnormality detection approaches have been proposed [1,6,11,3].Most of these approaches rely either solely on acceleration data (along the X and Z axes), or on various sensor fusion strategies.The acceleration based algorithms usually utilize some version of the so-called Gaussian background model in order to detect road surface abnormalities.In this work we use the algorithm introduced in [3] for automatic ground truth generation.The robustness of the method is well reflected in the fact, that it is even suitable for use when the vibration acceleration is measured by a mobile device.
Two measurements were used for our experiments.They were acquired using a modified Nissan Leaf test vehicle provided by SZTAKI.The measurements were recorded on the public roads of Budapest, with one measurement having been recorded on a newly built road and the other in an old parking lot. Figure 6 shows the labeled vertical acceleration data from the measurements.For the experiments described below we used the signals from the second bridge of the wheel sensor in the front right tyre of the vehicle, however preliminary investigations show similar results for all of the available signals.The measurements used contained roughly the same amount of normal and abnormal samples.The results show that each of the examined classifiers was successful in identifying abnormal road conditions from the wheel sensor measurements.The neural network based classification schemes required a fine tuning of hyperparameters, such as the number of layers and the number of neurons present in each layer.Finding the optimal hyperparameters was done via grid search.In order to ensure a "fair contest", the hyperparameters present in each of the classifiers were evaluated over the same search space.The convolutional networks and VP-NET configurations each contained a single convolutional/VP-layer.The specifics of each best performing network type are given in the below tables.The notation [a, b, c] means that the network consisted of 3 fully connected layers, with a, b and c neurons respectively.Based on the above results, VP-NET provides the most accurate wheel sensor based road abnormality detection.Furthermore the use of VP-NET is preferred because of the simplicity of the VP-layer compared to a convolutional layer, which makes it more suitable for low-level implementations.

Conclusion and future plans
In this manuscript we successfully demonstrated that accurate road abnormality detection based on signals from MFA's 3D force measuring sensor is possible.We presented an experiment that showed the connection between abnormal road conditions and the level of noise present in the residual signal (11).We then experimented with different classification schemes and found that VP-NET classifiers outperform the other candidates in both accuracy and simplicity.
There are several proposed next steps to continue this research.For example, ensemble methods could be created to further increase the accuracy of the classification.A VP-NET classifier could be used on signals from each bridge of the wheel sensor to identify road abnormalities, then some ensemble classification scheme could be applied to the output of the classifiers.A low level implementation of the classification schemes could also be created enabling on-line testing of the methods.

Figure 1 :
Figure 1: Signals produced by the wheel sensor.

Figure 3 :
Figure 3: The first few Hermite-functions.Their shapes closely resemble wheel sensor data.

Figure 4 :
Figure 4: Noise levels in the samples (top), and in the residuals (bottom).

Figure 6 :
Figure 6: Abnormal road surfaces detected in the acceleration data (red markings).Newly built road (left), old parking lot (right).

Table 1
shows the number of samples in each class.

Table 1 :
Sample numbers and ratio in each class.The different classification algorithms were trained on identical training sets and were evaluated on identical test sets.The training set contained 413 (80%) randomly selected samples, the remaining 104 (20%) samples were assigned to the test set.The examined algorithms were evaluated based on classification accuracy given as a percentage.The results are shown in table 2

Table 2 :
Examined classification schemes and their accuracy.

Table 3 :
Best performing convolutional network and training specifics.

Table 4 :
Best performing fully connected network and training specifics.

Table 5 :
Best performing VP-NET network and training specifics.