Contaminations on Lidar Sensor Covers: Performance Degradation Including Fault Detection and Modeling as Potential Applications

Lidar sensors play an essential role in the perception system of automated vehicles. Fault Detection, Isolation, Identification, and Recovery (FDIIR) systems are essential for increasing the reliability of lidar sensors. Knowing the influence of different faults on lidar data is the first crucial step towards fault detection for lidar sensors in automated vehicles. We investigate the influences of sensor cover contaminations on the output data, i.e., on the lidar point cloud and full waveform. Different contamination types were applied (dew, dirt, artificial dirt, foam, water, and oil) and the influence on the output data of the single beam lidar RIEGL LD05-A20 and the automotive mechanically spinning lidar Ouster OS1-64 was evaluated. The LD05-A20 measurements show that dew, artificial dirt, and foam lead to unwanted reflections at the sensor cover. Dew, artificial dirt over the entire transmitter, and foam measurements lead to severe faults, i.e., complete sensor blindness. The OS1-64 measurements also show that dew can lead to almost complete sensor blindness. The results look promising for further studies on fault detection and isolation, since the different contamination types lead to different symptom combinations.


I. INTRODUCTION
R ELIABLE perception sensors are a crucial step towards advancing the level of driving automation. Vehicles which provide level 3 "conditional driving automation", as defined by the Society of Automotive Engineers (SAE), must provide automated driving functions, which allow the driver to lift the focus from driving and only intervene when this is requested by the system. The responsibility of object and event detection followed by a proper response lies fully with the vehicle. Hence, perception sensors need to deliver reliable The review of this article was arranged by Associate Editor Abdulla Hussein Al-Kaff. results and report faults to the processing chain to reduce or revoke the trust in the results.
This requires a Fault Detection, Isolation, Identification, and Recovery (FDIIR) system that permanently monitors the perception sensors' performance for correct operation. The FDIIR system monitors the performance of the sensor constantly and triggers fault management. According to [1], actions in terms of fault management are: stop operation, change operation, reconfiguration, maintenance, and repair. Examples are wiper systems to remove dirt as described in [2] or switching to a safe state as described in [3] if a recovery is not possible. The development of such systems requires insight in how specific faults affect sensor performance. Therefore, extensive data collection and analytics studies are required as a basis for algorithm development. Data analytics is needed to investigate if and how faults are identifiable and whether their negative effects are critical for operation or only reduce the quality of results to an acceptable extent.
The perception sensor suite of an automated vehicle typically consists of camera, radar, and lidar. Each with their unique advantages and disadvantages which are mitigated by sensor fusion [4], [5]. Lidar is used to derive real-time 3D information of the physical world with a high level of confidence and is therefore an essential part of perception. A wide range of lidar systems exists. For an overview in the current state of the technology, see [6]. These lidar units can be compared by their most important specifications: detection range, transmitted power, wavelength, field of view, precision, accuracy, resolution, pulse rate, scan rate, and frame rate [7], [8].
Some of these specifications are impaired by faults, which can be classified into: defect subcomponents, mechanical damages to the sensor cover, contaminations on the cover, mounting related issues, security attacks, unfavorable environmental conditions, and sensor crosstalk [9]. Previously, [10] focused on dust layers, specifically on Arizona Dust, on the cover of a Velodyne VLP-16 where they found a detection range reduction of up to 75 %. Similarly, [11] investigated transmission and reflection of real world road dirt accumulated on the plastic cover with the aim of including the effects into simulations. In another study, a machine learning based algorithm for contamination classification outperformed classical approaches [12]. In addition to related work considering lidar, cover-related issues of automotive radar, especially water films on the radome, were investigated in multiple studies [13], [14], [15], [16]. We found no literature, however, that studies the effects of various contamination types on lidar point cloud and full waveform data. For this reason, our aim is to close this gap and focus on the influence of contaminations on the sensor cover, i.e., dew, dirt, artificial dirt, foam, oil, and water, on lidar point cloud and full waveform data. Our expectation is that this will be an important step towards fault detection, classification and choosing the right fault recovery method in order to follow the overall goal of reliable lidars.
The paper is structured as followed: Section II describes the measurement setup. Section III describes the methods and procedures for applying different contamination types and data analytics. Section IV presents the results and Section V discusses them. Finally, Section VI discusses those applications in which the observed effects may come into use and Section VII draws final conclusions and discusses potential applications for future research. Fig. 1 shows the measurement setup that was used for the experiments. It consists of the lidars which are mounted on a tripod and a target. The two lidars used are the RIEGL LD05-A20 (LD05-A20) and the Ouster OS1-64 Gen1 (OS1-64) which are explained in more detail in Sections II-A and II-B. The target used for the experiments is explained in Section II-C. The recording hard-and software are described in Section II-D. The measurements were taken on a rooftop at night to avoid any disturbances caused by sunlight.

A. FULL WAVEFORM LIDAR LD05-A20
The LD05-A20 is a full waveform lidar with a single beam. It provides information about the entire traveling path of the light. It thus supports the detection of multiple returns, i.e., multiple targets. The wavelength of the LD05-A20 light source is 905 nm [17].
We used an infrared camera for aligning the beam of the LD05-A20 to the target. For that purpose, we used a Sony SLT a-37 camera which was converted to a full spectrum camera by a company that is specialized on camera modifications. We added the Sony Alpha SAL1855 18-55mm F/3.5-5.6 DT objective to the camera. To filter light with a wavelength of lower than 720 nm, we used a 55 mm IR720 infrared filter. With the adjusted camera, the light of the LD05-A20 with a wavelength of 905 nm but also of the OS1-64 with a wavelength of 865 nm is visible.

B. AUTOMOTIVE LIDAR OS1-64
The OS1-64 is an automotive lidar with a mechanically spinning mirror and a horizontal field of view of 360 • . We recorded 2,048 points horizontally and 64 points vertically per rotation which is the lidar's maximum resolution. Therefore, one point cloud consists of 131,072 points. This results in 1,310,720 points per second using a sampling frequency of 10 Hz. The OS1-64 only captures the strongest reflection for each beam, i.e., a single point per beam. The wavelength of the OS1-64 light source is 865 nm [18].

C. TARGET
A 46 × 46 cm aluminum sheet mounted on a construction of aluminum profiles served as the carrier of our target material. In previous studies, we used retroreflective targets [19], [20]. According to the new datasheet of the OS1-64, the range accuracy is lower for retroreflective targets than for diffuse targets [18]. Therefore, retroreflectors are not optimal for the experiments and we used white paper as a diffuse target material which we attached to the aluminum sheet. As shown in Fig. 2, we compared the reflectance properties of white paper with a 10 %, a 50 %, and a 95 % Lambertian target by using a time-of-flight camera similar to our conducted measurements in [19] where we measured the reflectance of other target materials, e.g., retroreflectors and diffuse foils. We placed the camera in different angles from 0 • (i.e., orthogonally to the material) to 80 • to the materials. These measurements show that the course of the reflectance graph of white paper is similar to the graphs of Lambertian targets. White paper has a reflectance of 84.46 % at 0 • which means that white paper can be compared to a Lambertian target of 84.46 % reflectance. We used black absorbing tape to cover the aluminum profile. Therefore, the construction of aluminum profiles is not visible in the point clouds.

D. RECORDING HARD-AND SOFTWARE
We used a self-developed device for the power supply of the lidar sensors and for sensor data forwarding to a computer. The device includes a lithium-ion battery with 10 Ah and a Direct Current (DC)/DC converter, which provides a stable voltage to the lidar sensors. Furthermore, this device includes the interface printed circuit board provided by Ouster. We connected a laptop to our developed device to store the sensor data. The laptop uses the operating system Ubuntu 20.04 and the Robot Operating System (ROS) Noetic Ninjemys [23] as middleware to run the OS1-64 driver. The point clouds are stored in a .bag file, which is a file format provided by ROS. We use the web interface provided by the LD05-A20 to collect the data of the LD05-A20 full waveform lidar. The explained setup is the first version which is suitable for static measurements where point cloud and full waveform data are needed. In [24], we developed our setup further in order to support mobile and dynamic measurements, e.g., for mapping applications. The setup for mobile applications is called MOLISENS which stands for MObile LIdar SENsor System and integrates the OS1-64.
We recorded 450 point clouds per measurement for the OS1-64, which corresponds to a recording time of 45 s and 30 s per measurement with the LD05-A20.

A. FAULT INJECTION
We added an additional plastic cover to apply contaminations. In case of the OS1-64, the plastic cover is a cylinder to cover the lidar's horizontal field of view of 360 • . In case of the LD05-A20, the cover is a flat rectangle to cover the transmitter and receiver of the lidar similar to our previous study in which we applied mechanical damages [19]. In case of mechanical damages, we added the additional covers in order to protect the lidar sensors from damages. In the present study, we use it for two reasons: to protect the lidar sensor covers from damage, e.g., from dirt that scratches the cover, but also for exchanging the cover in order to avoid the influences of two different contamination types due to the contaminations of previous measurements. Furthermore, it is expected that lidar sensors will be integrated in the headlights of vehicles in the future as suggested and described in [25], [26]. Contaminations will then be on the cover of the headlights similar as it is in our setup. Fig. 3 shows the contamination types applied in case of the LD05-A20. Fig. 4 shows the contamination types applied in case of the OS1-64. We applied different types of contaminations to the sensor cover, including dew, dirt, artificial dirt, foam, water, and oil, in the following ways: • Dew: The dew point was at about 16 • C during measurements. We applied dew by putting the additional plastic cover into the freezer for about one hour. The sensor cover fogged up inside and outside when we took it out of the freezer.  that covers the entire field of view in the direction of the target. • Foam: We used a glass cleaner in a spray flask that sprays foam. • Water: We used a spray flask to apply water to the cover. • Oil: We used clean oil and applied it with a tissue to the cover.

B. LD05-A20 DATA ANALYTICS
We used the return detection provided by the RIEGL software V08Wave that detects the targets based on the peaks in the full waveforms. We extracted the range and the reflectance of far and near reflections for each full waveform. Furthermore, we counted the number of full waveforms received in 30 s of measurement time per experiment. In a next step, we calculated the distributions of the ranges of far reflections and the distributions of reflectances of near and far reflections for every experiment.

C. OS1-64 DATA ANALYTICS
We used the Python package pointcloudset for data analytics of the OS1-64 data stored in .bag files [27]. In a first step, we counted the point clouds with at least one point. We then filtered the point clouds, i.e., only points on the target remain in the point cloud and all the other points are removed. We counted the points that remained in the point clouds, i.e., the points on the target over time. Furthermore, we extracted the ideal target by using plane segmentation. Following on from this, we calculated the distances between the measured points on the target and the ideal target similar to our previous studies [19], [20].

A. LD05-A20
We counted the number of full waveforms that could be captured in 30 s for different contamination types as shown in Fig. 5. The reference without the additional plastic cover has 9181 full waveforms and the reference with the additional plastic cover has 9159 full waveforms. We added a horizontal line to the figure because this makes it easier to compare fault measurements with the reference. The water, oil, and artificial dirt measurements range between 9064 and 9225 full waveforms in 30 s of recording, i.e., contaminations of these types show similarity to the reference. However, the number of full waveforms is slightly reduced in the case of the five foam measurements which is between 8525 and 8964 and in the case of the five dirt measurements between 8005 and 9060. The biggest reduction in the number of full waveforms have the dew measurements which spread between 4274 and 4347 full waveforms in 30 s of measurement time. Fig. 6 shows the time series of measured distances of the LD05-A20. The reference scan shows that there is only one reflection at a distance at about 8.45 m. This is the distance between the lidar sensors and the target. We did not apply any faults, i.e., contaminations, to the reference scan. Therefore, no near reflections are observed ( Fig. 6 (a)). Also, the water ( Fig. 6 (g)) and the oil (Fig. 6 (f)), that we applied to the cover, lead to no near reflections. Therefore, oil and water do not reduce or block the view of the LD05-A20. Unlike water and oil, the dirt, artificial dirt, dew, and foam led to near reflections. In the case of dirt (Fig. 6 (c)) and artificial dirt of 20 × 20 mm (Fig. 6 (d)), the lidar has near reflections at  about 8 cm but also far reflections of the target. This observation was constant over time since the dirt did not move. In the case of dew ( Fig. 6 (b)), the LD05-A20 is completely blind over the entire measurement time, i.e., only near reflections and no far reflections are captured. In the case of foam ( Fig. 6 (e)), the foam blinds the LD05-A20 at the start of the measurements but this disappears increasingly over the 30 s of recording time. The time series of foam measurements show three areas of interest. The lidar is completely blind at the start. This is followed by a phase in the time series in which the lidar can see partly through the foam, i.e., near reflections and far reflections are observed at the same time. Finally, the lidar no longer has near reflections of the foam and it sees the target completely once again. The time of the transition between the phases varies over the five foam measurements. The transition between the first phase and the second phase is between 0.32 s and 7.02 s of measurement time and the transition between the second and the third phase is between 6.37 s and 24.12 s. Table 1 shows an overview of the observed effects. Fig. 7 shows that the median of the measured ranges, i.e., target distance, for the reference measurement is at 8.457 m. The first quartile is at 8.455 m and the third quartile is at 8.458 m. The median range varies little over the different measurements, so it is independent of the contamination applied to the sensor cover. However, it can be seen that the spread of the range values is higher if contaminations are applied, especially prominent is this observation for one experiment with dirt and for all of the foam experiments. These experiments show a range spread from 8.397 to 8.519 m, i.e., up to 0.122 m. The experiments with artificial dirt show that the spread of range values depend little on the size of the sticker applied, i.e., the spread is bigger and the target is seen closer, the bigger the sticker is. The dirt shows bigger spreads than the artificial dirt. Fig. 8 shows that the contaminations applied have an influence on the measured reflectance values of the target.  The reference measurements are close to 0 dB since white paper was used as the target material. Retroreflective targets have a reflectance higher than 0 dB. The reflectance of the target is negative when contaminations are applied. The reflectances of the foam experiments have a spread from approx. −20 to 0 dB, i.e., these experiments have a high spread of reflectances. Also, the dirt measurements show a large reduction of the target reflectance, especially the dirt measurement 5, which also has a big effect on the spread of the range. The results vary more for same contamination types than in Fig. 7. Fig. 9 shows the measurements for which near reflections at the sensor cover occurred and these are the dew, dirt, artificial dirt, and foam measurements. The reflection at the lidar cover, i.e., the peak in the full waveform, is prominent enough to detect it for these experiments. The foam measurements have the biggest spread in the reflectance values. The various dew measurements show almost the same distributions. The figure does not include the reference scan since no reflections at the sensor cover are detected when no contamination is applied.   Fig. 10 shows that there are no empty point clouds in the case of the reference measurement, all dirt, all water, and all oil measurements but also in the case of the artificial dirt with a size of 5 × 5 mm and 10 × 10 mm. For that reason, all of the 450 point clouds contain at least one point considering these measurements. Artificial dirt with a size of 15 × 15 mm leads to a reduced number of point clouds of 357 that contain points. In the case of foam, between 241 and 377 point clouds were received. Dew measurements lead to the biggest degradation of lidar performance since 0 to 77 point clouds include at least one point. Fig. 11 shows that the number of points on the target for the reference measurement is about 117. Only the oil measurements and the smallest artificial dirt of 5 × 5 mm show a similar number of points on the target. The biggest influence is observed for the dew measurements since the highest number of points on the target for some point clouds is 20. No points on the target were detected for dew measurement  3, artificial dirt of 20 × 20 mm, and artificial dirt over the entire transmitter. The figure also shows that the number of points on the target is lower, the bigger the artificial dirt is. It ranges between 0 and 10 for the artificial dirt of 15 × 15 mm. The foam measurements have a high spread of the number of points on the target. Fig. 12 shows that the reference measurement has a deviation of about 5 to 7 cm from the ideal plane. Most of the contamination types lead to higher deviations from the ideal plane compared to the reference. Only one dew and one oil measurement have slight lower deviations to the ideal plane. Dew measurements show the biggest spread of deviations to the ideal plane of up to 35 cm. It can also be observed that this spread gets bigger, in tune with the growth in the artificial dirt that is applied to the sensor cover. Bigger spreads of deviations are prominent for dew, dirt, artificial dirt, and foam measurements.

V. DISCUSSION OF RESULTS
All the contamination types were static, i.e., nothing moved during the measurements, except foam was running down the cover. For this reason, the foam measurements led to a bigger spread of various measured values as shown in Fig. 7, 8, 9, 10, 11: the reflectance and the range in the case of the LD05-A20 and the number of point clouds that are not empty and the number of points per point cloud in the case of the OS1-64. The layers on the lidar cover may also be influenced by wind or an airstream. For example, a wind or airstream may move, remove, or add contaminations to the lidar cover. This can lead to time-varying results comparable to the observations made in the case of the foam measurements. The variations between different dirt measurements shown in Fig. 7 and 8 are probably present because dirt was the most difficult contaminant to apply in a controlled manner, e.g., the size of dirt clusters and the number of dirt clusters varied from measurement to measurement. Also, the variations of the phase transitions in the case of foam shown in Fig. 6 are caused by the foam quantity since different foam quantities can cause different run down speeds. Although we observed sensor blindness over the entire measurement time in the case of dew, we made the visual observation that dew disappeared slightly from the cover over time. Fig. 13 shows multiple circular areas on the cover where dew disappeared, which seemed to be caused by the heat generated during lidar operation. For that reason, we expect that dew is only relevant in the first minutes of operation since it may disappear completely as a result of heat generated by the lidar but also from heat generated by the headlights in the case that lidars will be installed next to headlights in the future.

VI. DISCUSSION OF APPLICATIONS
The symptoms determined, i.e., the effects of sensor cover faults on the sensor data, can be used for developing lidar  fault detection methods which is described in more detail in Section VI-A, or also for lidar fault modeling, which is explained in Section VI-B.

A. LIDAR FAULT DETECTION AND REACTION
One possibility for fault detection is to calculate the moving average of an indicator and define thresholds for deciding whether a fault occurred or not similar to the stopping rule described in [28].
After detecting that a fault occurred, the contamination type may be determined by evaluating the combination of symptoms. The combination of symptoms on full waveform data for each contamination type is visualized in Table 2. These symptoms were evaluated visually based on the diagrams in Section IV-A. In our evaluations, water and oil show the same symptoms on full waveform data and are therefore hard to distinguish. The combination of symptoms on point cloud data for each contamination type is visualized in Table 3. These symptoms were evaluated visually based on the diagrams in Section IV-B. All of the contaminations show similar symptoms on the point cloud but of different severity. The severity of the symptoms still has to be evaluated for the classification of different contaminations based on point cloud data. Oil proved difficult to distinguish from the reference, since we did not find appropriate indicators. For that reason, it is expected that clean oil does not reduce the performance of the OS1-64. This may differ, however, for dirty oil.
Determining the type of faults may be relevant for choosing the appropriate fault reaction since different contamination types may need different cleaning methods for the sensor cover. An example may be the use of wipers as described in [2] appropriate for removing some types of contaminations that have a wide spread of measured values as observed for the foam measurements in the present work since these are probably contamination types that run down and are easy to remove. In contrast to that, dirt may need additional cleaning agents. Another fault reaction may be an adjustment in sensor fusion [4], [5], i.e., ignore or reduce trust in data of faulty lidars. In the case of sensor blindness, the sensor has to be ignored in sensor fusion until the cover is cleaned. In the case of partial sensor blindness, the sensor fusion algorithm may reduce the trust in the data of the faulty lidar until the cover is cleaned. A third fault reaction in the case of automated systems could be a change to a safe operation mode as described in [3], e.g., automated vehicles could drive slower or could be forced to stop on a safe place in case that the field of view is reduced.

B. LIDAR FAULT MODELING
To reduce the development effort of advanced driver assistance systems and to eventually enable automated driving functions requires the extension of conventional physical test drives with simulations in virtual test environments [29]. In particular, the perception system of an automated vehicle needs to be tested thoroughly in a very large number of different scenarios. In such a virtual environment, a perception sensor is simulated with a sensor model. Since automotive lidar plays an essential role in the perception system of automated vehicles [30], lidar modeling for virtual test drives is an important research field. Reference [31] provides a comprehensive overview of models for automotive perception sensors including lidar sensor models. Even though the effects of different sensor cover faults can have a strong influence on the perception capability of a lidar (as shown in this work), this topic has to our knowledge not yet been addressed for the existing lidar sensor models. Using a modular and configurable sensor model architecture as introduced by [32], the effects of sensor cover faults can easily be integrated into existing lidar models as additional modules.

VII. CONCLUSION
We observed that the contamination types applied, i.e., dew, dirt, artificial dirt, foam, oil, and water, have effects on the data of lidar sensors. All of these, except for oil, lead to changes of the analyzed OS1-64 indicators, i.e., number of points on the target and deviation between measured points and ideal plane. It would appear that full waveform information makes it possible to distinguish between contamination types according to a visual evaluation of the diagrams. Therefore, our experiments show that full waveform information is relevant for contamination classification which may be essential for choosing the right fault reaction. Oil has the smallest influence on lidar data. Therefore, we expect that the performance of the lidar will not be degraded by clean oil. Nevertheless, detecting and cleaning of oil films could also be important, since we expect that dirt will stick to the sensor cover more easily when an oil film is on the sensor cover.
In the future, fault injection may be standardized in order to compare the different lidar models from different manufacturers in terms of performance reduction in case of various contamination types. Here, we conducted five measurements per contamination in order to compare different contamination types and to investigate which contamination types are relevant for further investigations due to severe performance reductions. Future studies may focus on single contamination types by conducting more than five measurements per contamination type. Further contamination types may be ice, salt water, salt residues, or accumulated snow on the sensor cover. Moreover, future work may investigate the dependency of the results on target distance and material because targets in real world driving scenarios are at various distances and are out of different materials. Similar to a dirty windshield which reduces sight in case of a low standing and blinding sun, it would be interesting for further studies if sunlight is scattered more as a result of some contaminations, which could lead to an increased sensor blindness. Investigating the influence of wind and airstream on the results could also be considered in further studies since these are present at all times while driving. The symptoms of faults that we discussed are extracted by visually evaluating the diagrams. The symptoms may be investigated statistically in the future in order to provide reliable fault detecting and isolation.