Video Deraining using the Visual Properties of Rain Streaks

In computer vision applications, the visibility of the video content is crucial to perform analysis for better accuracy. The visibility can be affected by several atmospheric interferences in challenging weather – one such interference is the appearance of rain streaks. Recently, rain streak removal has achieved plenty of interest among researchers, as it has some exciting applications such as autonomous cars, intelligent traffic monitoring systems, multimedia, etc. In this paper, we propose a novel and simple method of rain streak removal by combining three novel extracted visual features focusing on the temporal appearance, wide shape and relative location of the rain streak. We called it the TAWL (Temporal Appearance, Width, and Location) method. The proposed TAWL method adaptively uses features from different resolutions and frame rates. Moreover, it progressively processes features from the upcoming frames so that it can remove rain in real-time. Experiments have been conducted using video sequences with both real rain and synthetic rain to compare the performance of the proposed method against the relevant state-of-the-art methods. The experimental results demonstrate that the proposed method outperforms the state-of-the-art methods by removing more rain streaks while keeping other moving regions.


I. INTRODUCTION
The visibility of a video is affected by many atmospheric interferences that degrade the quality of the video content. The video information is also affected by climatic events such as rain [1][2][3][4][5][6]. The low visibility degrades the performance of subsequent video analysis or processing applied in computer vision techniques. This undesirable situation degrades the performance of several computer vision applications such as driverless cars, intelligent traffic monitoring systems and surveillance systems [7][8][9]. As a result, it is necessary to improve the visibility of a video affected by external things like rain.
The scope of the paper is to remove rain streaks from video sequences. Fig.1 shows an example of the performance of the proposed TAWL method in generating a rain-free frame compared to the outcome using other existing methods.
(a) (b) (c) (d) FIGURE 1. Low visibility scenario due to rain and the performance of the rain removal algorithms to make a rain-free video where (a) a video frame with rain, (b) rain-free frame using an existing algorithm [18], (c) rain-free frame using another existing algorithm [19], and (d) rain-free frame using the proposed TAWL method.
Garg et al. [10] first raised a Rain Streaks Removal in Video (RSRV) method with a comprehensive analysis of the effects of visual properties such as spatial distribution, shape and velocity of the raindrops on an imaging system. They proposed that two camera properties, exposure time and depth of field adjustment, could reduce or even remove the effects of rain in a video sequence. Subsequently, many approaches have been recommended for the RSRV method and have achieved a good result in rain streak removal for a variety of rain conditions. Tripathi and Mukhopadhyay review wide-ranging primary video-based methods [11]. Chen and Chau study some very active scenes [12]. Kim et al. [13] have focused on the time-based relationship of rain streaks and the low-rank characteristic of rain-free videos. Santhaseelan et al. [14] marked and eliminated the rain streaks based on phase congruency features. You et al. [15] worked with environments where the raindrops were situated on window glass or car windscreens. Jiang et al. [16] focused on the directional property to propose a tensor-based RSRV method. Wang et al. [21] also proposed a tensor-based RSRV method using nonlocal low-rank regularisation. Ren et al. [17] worked with conditions of snow and rain. The authors considered a matrix decomposition technique. Wei et al. [18] have stochastically modelled rain streaks and have not considered the deterministic features. A mixture of Gaussian distributions has been used to model the rain-free background, while Li et al. [19] introduced the multi-scale convolutional filters from the rain data. Both methods reached a satisfactory level of performance with surveillance videos. Li et al. [24] have considered the dynamic indicator of both rain/snow and background scene. The authors encoded the rain streaks as an online multi-scale convolutional sparse coding (OMS-CSC) model.
The methods based on deep learning for the RSRV method also started to reveal their effectiveness [4,[25][26][27][28][29][30]. Most of the proposed models have been developed based on singleimage features but can also be applied to video sequences. A video sequence is a combination of single images called frames. These models have mostly addressed the interference/difficulties in visibility caused by the accumulation of rain streaks. Yang et al. [31] use binary mapping representing rain and without rain pixels to train the rain model. In the rain removal method, they used a contextualised dilated network. Liu et al. [29] have proposed a joint recurrent deep network for video deraining. A GAN based deraining method has been proposed by Matsui and Ikehara [1]. The authors have also introduced residual learning to develop a rain removal model. A two-stage RNN architecture has also been proposed for video deraining [27]. Also, a sequential deep unrolling framework has been proposed to exploit spatial and temporal features for video deraining [28]. Recently, some approaches have been proposed to combine feature based model and deep learning model [32,33].
However, some insightful characteristics possessed by rain streaks in a video still have not been explored. The literature identifies two difficult aspects of rain removal in video rain removal techniques: distinguishing rain streaks from moving regions and removing the rain streaks completely. The existing methods can perform better in one or the other aspect but not at the expected level in both aspects. Some of the existing methods cannot remove rain streaks in real-time for a video as they may need to store future frames to learn/train the models. Moreover, some existing methods cannot perform at the expected level if the frame rate and resolution of the video have been changed.
The insightful characteristics such as temporal duration, relative position within a frame, shape, location, etc., of the rain streak are crucial to distinguish the rain streak from the background and moving regions for making a video free of rain streaks. In this study, we propose a novel approach to remove rain streaks from a video to produce a better video by exploiting novel characteristics of the rain, such as appearance duration, shape, and location. The proposed method progressively learns the background, identifies rain streaks by utilising various features, and removes the rain streaks frame by frame. Furthermore, as our method extracts features for every current frame that is being processed, it can be applied in real-time application considering some processing time.
First, we modelled the background with Low-Rank Matrix Factorisation (LRMF) and applied a Mixture of Gaussian (MoG) to separate the background and foreground [19][25] [33]. After separation of the background, the foreground usually includes any rain streaks and moving objects. The main challenge of the rain removal algorithms is to separate the rain streaks from the moving object for the rain-free video. Here we model the rain streaks based on the temporal appearance (TA) of the rain streaks in the video. We have observed that rain in identical locations in head-tohead frames of a video sequence is highly unlikely. We have considered this property to distinguish rain streaks from other moving objects, bearing in mind the impact of the frame rates of the video. But it misses some portions of moving objects that also have a shorter appearance. To solve this issue and recover the missing moving objects, we extract more features based on the other properties of rain streaks. We exploit the width of the rain streak, which filters some false positives from the candidate rain pixels. We have observed that rain streaks have a range of specific widths, bearing in mind the resolution impact. Combining TA and width properties provides a good rain removal performance; however, some false positives are still detected. The candidate pixel's location is also a significant property to distinguish rain streaks from moving objects. We also exploit location-wise properties to identify rain from other moving regions.
This paper proposes a novel algorithm by combining the TA properties of rain streaks with the shape and location properties of rain streaks to improve the recovered moving objects in rain-free videos. Please note that we identify each pixel with rain or without rain considering the temporal correlation of that pixel and relative position of the pixel against neighboring pixels, whereas the algorithm in [10] considered directional correlation of the rain streak by considering a number of pixels. We avoid explicit rain streak direction as the rain streak's direction can be aligned with the moving object's movement direction. This may lead to a wrong prediction of the rain. We also see the performance of the proposed method in different resolutions and frame rates as the TA, shape and location properties have been changed with frame rates and/or resolutions. We have adaptively used the properties criteria to make the proposed method effective in different resolutions and frame rates.
The preliminary idea based on the TA is published in a conference paper [7]; however, our contributions are: • We introduced and formulated the temporal appearance (modified compared to [7] in the light of frame rate invariant) of rain streaks to differentiate them from other moving objects. • In addition, we also developed and formulated the two other vital features of rain streaks based on the shape and location where the correlation of the neighboring pixels of an already identified rain streak has been exploited. • We have fused different criteria to make the final decision. We have adaptive thresholds to make the proposed method effective in different resolutions and frame rates. • We have compared the proposed method with different approaches like deep learning-based model, self-learning model and feature-based model. We have also implemented proposed method on few more video sequences which include different rain streaks pattern and environmental condition. These comparative results and analysis are not included in the conference proceedings. The rest of the paper is organized as follows: Section II describes the proposed method and justifies and explains every step. Section III contains the setup of the experiment and analysis of the results, and Section IV concludes the paper

II. PROPOSED METHOD FIGURE 2. Block diagram of the proposed TAWL rain streak removal method.
The proposed rain streak removal algorithm contains five significant steps (see the schematic diagram in Fig. 2): (i) background and foreground extraction, (ii) TA feature of rain streaks, (iii) width feature of rain streaks, (iv) location feature of rain streaks, and (v) rain-free video generation. The red marked block indicates two added features which is an extended contribution to our preliminary work published in conference. In this method, we explore insightful rain streak properties to refine the candidate rain streak pixels. First, we separate all moving regions, including rain and moving objects, using the generated background frame from a current frame. The background frame is generated from the past frames of a video using an existing method [19][25] [33]. To separate the rain streaks from other moving regions, we exploit the TA property of rain streaks, including some briefly appearing moving regions with rain streaks. Then we apply the width property to filter out moving regions that are brief appearing but relatively bigger in size as the rain streaks usually are smaller in width. Finally, we apply the location of candidate pixels to filter out the false-positive selection of rain streaks as the rain streaks are generally isolated and scattered. All the five significant steps are discussed in the following sections in details. Note that here input frames are in YCbCr color format and the processes are applied on the Y component of the input frame.

A. BACKGROUND MODELING
Many dynamic background modelling approaches [34][35][36] are available and the basic concept to develop these models is very similar. The background remains the same over all the frames in a video scene captured by a static camera except for the interference of moving objects and change of light. Thus, this background layer can be formulated as recovering a low-dimensional subspace [37][38][39][40][41]. The regular approach to subspace learning is the subsequent low-rank matrix factorisation (LRMF): where, ∈ × , ∈ × , = ℎ , < min( , ). The operation of 'Fold' refers to folding up each column of a matrix into the corresponding frame matrix of a tensor. At each frame, we generate a background frame. We use the background frame to find rain streaks and other foregrounds to generate the rain-free video in the proposed method.
Initially, we have generated the foreground by subtracting the background from the input frame, where F is a foreground binary image of the n th frame, In is the original n th frame and Bn is the background frame at the n th frame. Here we use an intensity threshold value of 20 to eliminate the effect of other light or illumination interference from the generated foreground. This image contains rain streaks and moving objects.

B. TEMPORAL APPEARANCE (TA) WITH FRAMES RATE INVARIANT
After subtracting the background frame from the current frame, we can get the foreground, which comprises rain streaks and moving objects. Fig. 3(a) shows the original frame 85, Fig. 3(b) shows the Y component of the original input frame. Fig. 3(c) shows the background frame of frame 85 and Fig. 3(c) shows the rain streaks and moving objects of frame 85 in greyscale for the Traffic video sequence. The figure demonstrates that the background modelling with the threshold successfully detected moving regions, including rain streaks. To separate the moving objects from the rain streaks, we exploit the TA property of the rain streaks in a video sequence. We have observed that the appearance of the rain streaks in the same location in adjacent frames of a video sequence is highly unlikely. We have exploited this property to separate rain streaks from other moving objects. The red circles marked in Fig. 4 demonstrate the TA property of the rain streaks. Two adjacent frames (frame 84 and frame 85) represent rain streaks in four locations of each frame. Rain streaks appear at two red circles (i.e., top and top right) in frame 84 but disappear in frame 85. Rain streaks do not appear at two red circles (i.e., bottom and left bottom) in frame 84 but appear in frame 85. This observation demonstrates that the rain streaks appear at a particular location of a frame in a video very briefly and may comprise a few frames depending on the frame rate of the capturing devices. However, the moving object normally does not show low appearance characteristics like rain streaks in an area. Rain streaks appear in a video discreetly; normally, they change location frequently for low to mid-intensity rain.
In comparison, moving objects change location smoothly (see the moving car). We have used this temporal appearance characteristic of rain streaks to separate the rain streaks from the moving objects of the foreground. To model the rain streaks and analyze their temporal characteristics, we created a mask that uses the binary image F of each frame for the previous m frames adjacent to each other. In a binary image, a "1" represents the foreground, including rain and other moving objects, and a "0" represents the background. where M represents a mask of the n th frame, F represents the foreground binary images of adjacent frames (described above), and m is the maximum number of adjacent frames. We use the previous m number of frames to make the decision contemporary, as the scene may be changing significantly enough so that the mask may not be relevant to represent the recent changes. We consider the appearance value of every pixel location in the mask. If the appearance value is more than a certain duration threshold in terms of the frame rate of the video, it is considered as the part of the object area and any value more than zero and up to that duration threshold is considered as the rain area; otherwise, it is considered as part of the background area. We use the duration threshold as 20% of the frame rate to classify the rain, object, and background areas as the appearance duration vary with capturing frame rates (see explanation below). In Fig. 5, all the yellow area is considered as the object area, the red area as the rain streaks area and the blue area as a background area. The TA value mostly depends on the frame rate because if the capture device is operating in a high frame rate, a rain streak may appear in a greater number of frames. That's why for better rain removal, we need to make the proposed method invariant of frame rates or, in other words, the proposed method should be applicable for different frame rates. Thus, the threshold we have used against the mask M is a function of the frame rates so that the threshold can be adaptive with the frame rate for successful rain removal. Fig.  6 shows the effect of different frame rates of a video if we use a constant threshold value in different frame rates. For the video with higher frame rates, the loss of moving objects is less compared to that of lower frame rates. The results are different for different frame rates of the video with a fixed threshold. Thus, we can successfully exploit the TA property using an adaptive threshold for different frame rates. The rain-free frame and the rain streak detected using only the TA property are shown in Fig. 7. (a) At a lower frame rate (b) At a higher frame rate FIGURE 6. The effect of the different frame rates on TA output with a constant threshold value of TA modelling using the "Traffic" video sequence.

C. WIDTH AND LOCATION PROPERTIES FOR FRAME RATE AND RESOLUTION INVARIANT
After applying the rain streak model based on the TA feature of rain streaks, we missed a few portions of a moving object in the rain-free video frame as the model misclassifies some parts of the object as a rain streak (Fig. 7). For example, a portion of the moving car is also identified as a rain streak in the TA process (see Fig.7 (c)), together with the rain streak. As a result, the separated object is distorted (Fig.  7(d)), which affects the rain-free frame (Fig. 7(a)). To overcome this issue and reduce the false positives due to the inclusion of moving objects in the outcome of the TA process, we need to filter out the moving objects from the identified rain streaks using other properties of the rain streaks. We refine the candidate rain streak pixels by two consecutive filters. Before applying these filters, we created two binary images for each frame based on TA properties. One consists of candidate rain streaks and the other includes moving objects. We modelled both filters based on rain streak characteristicsthe rain streak's width and the relative position of the candidate pixels. They are discussed in detail below:

RAIN STREAK WIDTH
From the candidate rain streaks, we filtered out the false positives by measuring the rain streak width. The rationale for using the width of the rain streak is that the width is not very wide and normally discrete. We check the number of consecutive 1s in every row of the binary image that includes candidate rain streaks. In this filter, we consider a length threshold with respect to the frame width as a maximum rain streak width. Thus, any rain streak candidate with a number of consecutive 1s in a row that is less than the length threshold of the frame width is considered a rain streak. We use 5% of the frame width as the length threshold. We experimentally observed many datasets with different types of rain streaks and found our consideration works better. We consider the threshold against the frame width to make the filter resolution-independent, so the adaptive threshold should work in different resolutions. The pixel value of rain streak width is highly dependent on the image resolution. We experimentally observed the effect of different resolutions of a video sequence. Fig. 8 shows the results with different resolutions of a video sequence where the threshold of rain streak width is considered constant. The figure demonstrates that the rain streak width filter missed some rain streaks for higher resolution video by considering them as objects, as their pixel size is increased due to the larger resolution. On the other hand, the rain streak width filter can select more rain streaks at a lower resolution.

RAIN STREAK LOCATION
After refining the width property, some false positives are filtered out; however, some false positives remain as it includes some moving regions with a similar width to the rain streaks. To further filter out the moving areas, we check the neighbourhood pixels of the candidate rain streaks to determine how close they are to the moving objects. If they are very close and connected with the moving objects, they are considered part of the moving objects rather than the rain streaks. Thus, they need to be filtered out as false positives. The neighbour pixels of each candidate pixel is checked using the separated binary objects image. The same number of neighbour pixels in each quadrant (up-right, up-left, down-left and down-right) of the candidate pixels are checked to see whether they are an isolated cluster of 1s or not. If they are an isolated cluster of 1s, then we assume that they are a rain streak; otherwise, they are a part of a moving object. The rationale of this assumption is that if the cluster of 1s has connected with another cluster of 1s, then they are part of a moving object rather than a rain streak as the rain streak is usually isolated. Fig. 9 shows the results and comparative improvement in outcomes after applying the width and location features with TA features. Fig. 9(a) & (b) show that the object is well recovered after applying the width and location features, where it is distorted after applying only the TA features. Likewise, the separated rain streaks are improved after applying the width and location features. The distorted object portions are selected as rain streaks after applying only the TA features (Fig, 9(c)), whereas no object portion is selected as a rain streak in Fig 9(d).

D. RAIN-FREE VIDEO GENERATION
After applying all those extracted features, we have generated an object mask for the current processing frame. We have used both the generated background frame at the current frame position and the current frame to generate a rain-free video frame. For example, we use both the ith background and the ith frame to generate the rain-free frame for the current i th frame. We identify each of the pixels as a background, rain or moving object through processes mentioned earlier. For a rain-free frame, if the pixel is identified as background or rain, then the corresponding pixel intensity is taken from the background frame, and if the pixel is identified as a moving object, then the corresponding pixel intensity is taken from the current frame. Fig. 10 shows the outcomes after applying all steps.

III. EXPERIMENTAL RESULTS
We have conducted experiments using video sequences with real rain to compare the performance of the proposed method and other contemporary and relevant methods. This comparison provides a subjective quality assessment as there is no ground truth for the real rain-free videos. We also compare the performance using video sequences with synthetic rain to understand subjective and objective measurements, as the synthetic video sequences have a ground truth for comparison. We have not compared the time cost of different methods here. The compared methods (i.e., PMoG, MS-CSC) extract features from all frames together, where the proposed method uses just the current frame and few previous frames to extract features so that the proposed method can be used in real-time applications. The features extraction process is not the same. Thus, comparing time costs will not represent a fair scenario.
We have considered six methods to compare the performance of the proposed method including three modelbased video deraining methods PMoG [18], MS-CSC [19], and OMS-CSC [24] and three DL-based video deraining methods SPAC-CNN [25], SLDNet [42], and S2VD [30]. These methods are relatively recent and work on the RSRV; thus, we select these six methods to compare. The feature extraction with physical meaning gives us a better understanding of the rain characteristics for classification. Fig. 11 shows the experimental results of the "Traffic" video sequence at frame 72. The figure demonstrates that the proposed method outperforms the state-of-the-art methods in both rain removal and object retain cases. The identified areas using green rectangles show some distortions in the MS-CSC [19] and OMS-CSC [24] method's results. This portion is a part of an object reflection. Moreover, the proposed method successfully removes more rain streaks compared to other methods by maintaining a better quality for moving regions.

Input frame
PMoG [18] MS-CSC [19] OMS-CSC [24] Proposed FIGURE 11. Rain removal results and comparison between the proposed method and other methods for the video sequence "Traffic". Fig. 12 shows the results of the "Yard" video sequence at frame 97. The proposed method performs better than the other methods. It has removed more rain streaks than the other methods. The rectangle and circle marked areas clearly show that the proposed method can remove more rain streaks.

Input
PMoG [18] MS-CSC [19] Proposed FIGURE 12. Rain removal results and comparison between the proposed method and other methods for the "Yard" video sequence. Fig. 13 shows the results of the "Wall" video sequence at frame 15. The proposed method outperforms the other two methods. It has removed more rain streaks and generated a high-quality rain-free video frame compared to the other methods [18][19].
input PMoG [18] Fig. 14 shows the visual comparison of different methods on frame 35 of a real testing video sequence called "Ra4" from dataset NTURain [25]. The proposed method outperforms the other two methods. The selected area shows that the rain-free frame is cleaner in the result for the proposed method.

Input
PMoG [18] MS-CSC [19] Proposed FIGURE 14. Rain removal results and comparison between the proposed method and other methods for a video sequence called "Ra4".from dataset NTURain Figure 15 shows the visual results of different methods on frame 93 of a real video sequence called "Saigon" from a dataset MS-CSC [19]. The proposed method outperforms the other methods to remove the rain and also retains the moving objects. However, the MS-CSC method fails to retain some parts of the moving object (see the highlighted area).

B. EXPERIMENTAL RESULTS OF SYNTHETIC RAIN STREAKS
To understand the performance of the proposed method compared to the other two methods, we also provide experimental results using videos with synthetic rain. Fig. 17 shows the results of a synthetic video sequence called "Truck" at frame 65. The proposed method can remove almost all rain streaks, while the other methods fail to remove rain streaks in several areas.

Input
Ground truth

PMoG [18] MS-CSC [19]
Proposed FIGURE 17. Rain removal results and comparison between the proposed method and other methods using a video sequence called "Truck". Fig. 18 shows the results of another video sequence called "Park" at frame 124. This video has synthetic rain. The visual result shows the proposed method and MS-CSC [19] perform very well in rain removal, whereas PMoG [18] is not as good as the proposed method. The identified areas using red rectangles show some distortions in the results of MS-CSC [19] and OMS-CSC [24]. This portion is a part of a moving man's leg. Moreover, the proposed method successfully removes more rain streaks compared to the other methods by maintaining a better quality for the moving regions.

Input
Ground truth   Figure 20 shows the quantitative comparison of the proposed method against the other two methods using the video sequence "Truck" in terms of the PSNR value in each frame. In the figure, the input PSNRs mean the PSNRs of the original frames against ground truth frames (i.e., without rain), which should be the lowest as they have rain. The proposed method outperforms the method in [18] for all frames. It outperforms the method in [19] in most of the frames. This demonstrates that the proposed method successfully removes rain from the frames and maintains a better quality for moving objects. Table 1 shows a comparison of average PSNR values of all frames for the two synthetic datasets, "Truck" and "Park".
For both datasets, the proposed method performs better than the other methods.

IV. CONCLUSION
This paper tries to understand the insightful characteristics of rain streaks and then use them to make a rain-free video. To do this, we identify three crucial characteristics: temporal duration appearance, width, and relative location of the rain streaks. The appearance of the temporal duration is an important phenomenon of rain streaks, as a rain streak lasts in a pixel location for a short time, i.e., for a few frames. The rain streak has a certain width for low or medium density rain. Moreover, the location of the rain streak is naturally scattered or isolated. We use dynamic background modelling to gradually exploit these features to identify rain streaks from normal moving regions after separating the rain and moving regions. We also process the features in such a way that they can be applicable for different resolutions and frames of the video sequences. Moreover, we also extract the features and use them in such a way that the proposed method can work in real-time. To verify the superiority of the proposed method, we use video sequences with both real and synthetic rains and compare the performance against two contemporary and relevant methods. The experimental results confirm that the proposed method outperforms those methods in terms of better rain-free video and higher quality moving regions. In the proposed method we did not consider heavy rain explicitly. In our future works, we may consider different types of rains e.g., heavy rains, light rains, etc. and make the proposed method adaptive to provide a unified solution for rain removal.