Improving Post-Processing of Audio Event Detectors Using Reinforcement Learning

We apply post-processing to the class probability distribution outputs of audio event classification models and employ reinforcement learning to jointly discover the optimal parameters for various stages of a post-processing stack, such as the classification thresholds and the kernel sizes of median filtering algorithms used to smooth out model predictions. To achieve this we define a reinforcement learning environment where: 1) a state is the class probability distribution provided by the model for a given audio sample, 2) an action is the choice of a candidate optimal value for each parameter of the post-processing stack, 3) the reward is based on the classification accuracy metric we aim to optimize, which is the audio event-based macro F1-score in our case. We apply our post-processing to the class probability distribution outputs of two audio event classification models submitted to the DCASE Task4 2020 challenge. We find that by using reinforcement learning to discover the optimal per-class parameters for the post-processing stack that is applied to the outputs of audio event classification models, we can improve the audio event-based macro F1-score (the main metric used in the DCASE challenge to compare audio event classification accuracy) by 4-5% compared to using the same post-processing stack with manually tuned parameters.


I. INTRODUCTION
A S it is common with most audio analysis methods that operate by taking classification decisions at the audio frame level, the goal of a final post-processing stage is to increase the classification performance of an inevitably imperfect model by smoothing out the sequence of predictions, removing classification noise like false-positive or falsenegative decisions and improving the boundaries of detected audio events. A typical post-processing stage consists of a soft output thresholding step followed by a smoothing operation, with median filtering being a popular choice. The parameters of such operations are usually hand-crafted following some grid search technique. It can therefore be understood that post-processing schemes have received significantly less attention over recent years, compared to the core audio classification algorithms. To remove the need for such manual parameter tuning, we propose a reinforcement learning [1] approach that automatically computes parameter values of the post-processing pipeline operations by treating them as learnable parameters that can be applied to the class probability distribution outputs of audio event detection and classification models. A widely accepted problem formulation for common audio event detection and classification problems can be found in the DCASE Task4 challenge task descriptions, where, for example, according to the sound event detection task specification (Task4 in recent years) [2], a model must be able to identify the endpoints of each audio event in a given audio recording along with the class label of each audio event. Examples of such audio events are human speech, dog barks, vacuum cleaner sounds and so on. In general, audio events may overlap and multiple classes are expected to be accommodated by an algorithm (10 classes is a typical scenario). Following the DCASE Task4 challenge practice, audio event detection algorithms FIGURE 1. Illustration of the proposed MDP approach to optimize the parameters of a post-processing stack for an audio event classification model. The RL agent receives the class probability outputs of the model and chooses the parameters per class for each part of the post-processing stack (in this case thresholding and median filtering). The post-processing stage with the given parameters is subsequently applied to the model outputs and the F1-score is updated. The agent then receives a reward based on the achieved score and proceeds to the next audio segment.
are primarily evaluated based on the so-called event-based macro F1-score [3], according to which a certain tolerance is permitted around the detected event endpoints, like a 200 ms tolerance collar on event onsets and a 200 ms (or 20% of the events length tolerance) collar on event offsets. In order to validate our approach, we apply the proposed RLbased post-processing method to the class probability outputs of two audio event detection models that were submitted to Task 4 of the DCASE-2020 challenge: a) A baseline detector which was developed by the organizers and achieved a 40.1% event-based macro-F1 score [2] on the evaluation set of the challenge and b) The best performing detector of the 2020 challenge, ConformerSED [4] which achieved a 47.7% F1-score on the same evaluation set. Both systems apply class probability thresholding and a subsequent median filtering step at a post-processing stage and are, therefore, good candidates for our method. In both challenge submissions, the respective parameters were manually tuned by the method proposers using a coarse grid search in the parameter space and were class-independent, i.e., the same values were applied to all classes. If, instead, our RL-based method is used to automatically fine-tune a classification threshold and median window length per audio event class, the F1-score of the baseline method increases to 44.5%, and the F1-score of the best performing method increases to 52.4%. We therefore observe a 4% to 5% performance improvement on average, over very different methods, if the parameters of a standard post-processing pipeline are tuned using RL, assuming that the RL agent has access to the soft outputs of a method for a set of training samples.
Our contribution can be summarised as follows: • We introduce a process based on reinforcement learning for optimizing the parameters of the typical postprocessing stack (thresholding and median filtering) applied to the prediction outputs of an audio event detection system.
• We demonstrate that this reinforcement learning optimization approach of the post-processing parameters provides superior audio event detection performance compared to the typical grid search optimization approaches. The code for this work is available on GitHub 1 .

II. RELATED WORK
The work in [5] conducted an evaluation of post-processing algorithms that were applied on the output of the baseline DCASE-2018 Task4 audio event detector model and another submitted model. They concluded that post-processing has a large impact on the final classification performance as measured by the event-based F1-score and that postprocessing algorithms performed better when they were class-dependent. The authors in [6] proposed a frame-wise dynamic threshold selection method for polyphonic sound event detection and showed that it was superior to using a fixed threshold for all audio frames. To estimate the threshold per frame, they used contour-based and regressor-based methods. Our method adds to the post-processing approaches outlined in these previous works by introducing RL as a way of optimizing the parameters of post-processing, as far as both thresholding and median filtering are concerned. Overall, however, the optimization of the post-processing stage of audio event detection systems has not received enough attention in literature so far, in our opinion, while it has been shown that it can have a significant impact on the final performance of audio event detection systems [5]. Regarding RL methods for audio analysis tasks, recent systems primarily address the application of autonomous agents that are capable of leveraging auditory information to perform diverse tasks given an environment, e.g., [7]- [11], but, to the best of our knowledge, our work is the first to formulate parameter tuning at the post-processing stage as a RL task and employ a RL algorithm to automatically compute parameter values for improving the final classification performance of audio event detector systems.

A. POST-PROCESSING FUNDAMENTALS FOR AUDIO EVENT DETECTION METHODS
Given an audio recording, audio event detectors usually perform strong labeling or weak labeling tasks. The former refers to problems where both endpoints and class labels of audio events need to be reported, whereas the latter refers to tasks where only the presence or absence of an audio event type needs to be returned, thus skipping the need for endpoint detection. Note that it is common to assume that in a multi-class, strong labelling context, two or more events may overlap. As it is usually the case, a soft decision output is computed per class over the duration of the recording and a threshold is subsequently applied for each event class. Soft values above the threshold indicate class presence in the respective frames. For the strong labeling task, when successive frames yield values above a threshold, they are grouped to form an event. In the case of weak labeling problems, thresholding is used to identify which event classes are present in the audio segment by collecting a percentage of frames that pass the threshold test per class.
As an example, if a detector model outputs posterior class probabilities at the frame level or at the audio segment level, a threshold of 0.5 means that an audio event is considered to be present if the respective class probability is higher than 0.5. A straightforward threshold selection approach has been to set a fixed threshold value for all classes. This type of naive threshold selection may work on certain occasions but it will fail to provide good results for many distributions of model outputs over the audio event classes ( Figure 2) [6]. We can also set different thresholds per event class, which can potentially take into account variations in the model's behaviour for different event types. Apart from setting thresholds on the dataset level, we can increase threshold selection granularity and set thresholds at the audio clip level, or, in the case of strong labels, even at frame level. Our work computes a threshold per class at the audio clip level and it is suitable for both strong labeling and weak labeling tasks.
Median filtering is a second post-processing step that is used especially in strong labeling tasks to improve event detection accuracy. It can remove certain artifacts of the event detector, like false positive or false negative decisions that only last a few successive frames and can therefore smooth out the detected audio event boundaries ( Figure 3) and reduce over-segmentation. The median window size can play an important role and different sizes may be appropriate for various types of events [5], in order to balance undesirable (excessive) smoothing of event boundaries and decision noise suppression. Strong (frame-level) labeling of audio events at the output of an AED system, taken from [6]. Global fixed thresholds can lead to missed events and wrong classifications.

B. POST-PROCESSING OPTIMIZATION USING REINFORCEMENT LEARNING
Based on the above description, a standard post-processing pipeline of an Audio Event Detector (AED) system performing strong labeling consists of thresholding and, optionally, median filtering. The problem is to estimate threshold values that strike a balance between precision and recall for each class, as well as median window sizes that remove detection artifacts without causing excessive smoothing while taking into account the characteristics of the system response for each audio event class.
Our method assumes that there exists a set of reasonable values for each parameter of the post-processing pipeline, e.g. a (0.0, 1.0) continuous range for the threshold selection step and discrete [3,5,7, . . . , 21] values for the median filtering window size. Contrary to the range of threshold values, the range of median filtering window sizes is unbounded. For this reason, we had to select a range of median filtering window sizes that are sensible and most commonly encountered in AED systems literature. Our goal then becomes to find "optimal", class-dependent parameters for each audio segment in the dataset. We can therefore consider the problem VOLUME 4, 2016 Parameter Value sample rate 16000 max audio segment length (seconds) 10 number of classes 10 threshold range (0.0, 1.0) set of median filtering window sizes [3,5,7,9,11,13,15,17,19, 21] from a MDP perspective, i.e., as solving a MDP (Figure 1), where a state S is the AED output for a given audio segment, i.e., a matrix of class probabilities, one row per frame, like in (Figure 2). Given S, we select a threshold value and a median window length from the respective sets of possible values. If we are dealing with a class-dependent optimization scenario, two different values will be selected per class. Therefore, the action vector, A, of the RL agent, will consist of 2 × N elements, where N is the number of classes. If we are interested in a simpler, class independent setting, there will only be two values per audio clip and the action vector of the RL agent will therefore be two-dimensional. Then the processing stack with the selected parameters is applied to the AED outputs and the event-based macro F1score for that audio segment is evaluated. The higher the F1score, the higher the reward, R, of taking action A in state S. The goal is to maximize the average reward over the entire dataset, which directly correlates to maximizing the average F1-score. Given that the F1-score is in the range [0%, 100%], and the reward should be normalized in the range [0.0, 1.0], for training stability purposes, it follows that the relationship between the reward R and the F1-score would simply be R = F 1/100.

C. POLICY GRADIENT OPTIMIZATION
To solve the MDP we employ Policy Gradient (PG) reinforcement learning method [12], [13], where the policy function approximator is a simple recurrent neural network made up from a stack of 2 GRU layers [14] (Figure 4). We chose a recurrent architecture in order to be able to better capture time-dependent information found in the sequences of predictions of an AED system, given that the AED system itself usually performs classification over audio signal feature sequences that it receives as input. The objective of the PG algorithm is to find a policy function π θ , where θ are the weights of the neural network, which adequately approximates the optimal policy π * . When π * is followed it yields the maximum achievable mean reward R over the dataset, which translates to the maximum achievable eventbased macro F1-score. The PG algorithm iteratively makes π θ converge to π * by assigning a value to each state S, which represents the average reward that it expects to get in the future by following the current policy π θ in S, and it is directly related to the average expected F1-score. More formally: where t ∈ [0, T ], with T = len(dataset), is the iteration index, r t is the reward at iteration t, and γ ∈ (0, 1] is the future reward discount factor, which governs how much "weight" the algorithm should place on long-term vs shortterm rewards. After the value of state S is calculated, the next step is to estimate the value (or advantage) of taking action A in S at iteration t as the sum of the expected values of future states, resulting from taking action A in state S, denoted here as s t and a t : where At each iteration t the policy approximator network outputs log-probabilities (after the application of log-softmax) for each action, multipled by the estimated advantage of that action: log π θ (a t |s t ) The network outputs two actions a t given state s t . In the case of class-dependent optimization each a t is a vector of log-probabilities, of length nClasses. Each vector entry is another vector of length nV alues, where nV alues is the number of possible values for the threshold and for the median filtering window size. In the class-independent case a t is simply a vector of length nV alues containing scalars.
Since threshold values are continuous, they are sampled from a uniform distribution in the range (0.0, 1.0).  TABLE 3. Event-based macro F1-score for both Baseline and ConformerSED systems, when the original parameters for thresholding and median filtering window are used, and when optimized via Policy Gradient reinforcement learning algorithm. We try both class-dependent and class-independent optimization. We also compare the performance when only thresholding is used and when both thresholding and median filtering post-processing are used. The results are obtained on the validation set provided by the DCASE Task4 2020 challenge.
After multiplying the action log-probabilities (4) with the advantage estimate for each action (2), the final objective optimized by PG for a batch of log-probabilities is: where E is the mean over the batch. To promote exploration of the state space and avoid convergence to a sub-optimal policy, an entropy regularization term H(θ) = −π θ log π θ is added to the PG objective [15] so that the final objective becomes: where β is the weight of the entropy regularization term in the final objective. In place, or in addition to entropy regularization, other methods for facilitating exploration of the state space could be used, such as the addition of gaussian or uniform noise to the continuous threshold predictions of the network or randomly replacing the discrete median filtering window size predictions of the network with other values sampled from the set of window sizes.

IV. EXPERIMENTS
Similar to [5], we apply our method to the optimization of the post-processing stack of two AED systems: a baseline approach and a state-of-the-art method, to cover the two extremes of the performance spectrum. Table 1 shows the values of the parameters that define the state and action spaces of the reinforcement learning environment on which the Policy Gradient algorithm is trained. Specifically, the state space is defined by the sample rate, maximum length of audio segments and number of classes present in the dataset. The sets of possible threshold values and median filter window sizes define the action space. Table 2 shows the values of the Policy Gradient algorithm hyperparameters that are used for training. Specifically, the choice of batch size was constrained by the available system memory, or how many copies of the dataset could fit in memory. Memory size refers to the history of training iterations, or interactions of the RL agent with the MDP environment, that are buffered and is also constrained by available system memory. The rest of the parameters, such as the choice of optimizer, learning rate, update frequency were taken from [15]. The training parameters remained identical for both AED systems that the PG algorithm was trained on. The most important ones are the discount factor (γ) and the entropy regularization weight. A discount factor close to 1 is needed to assign more importance to rewards gained in the future, because the F1score is only calculated once at the end of the dataset after the predictions for all audio segments have been accumulated. The entropy regularization weight value achieves a balance between exploration and exploitation: a low value leads to faster convergence but less exploration of the state space, which in turn can lead to convergence to a sub-optimal F1score.
The results of our experiments are grouped in Table 3, for the two AED systems that we evaluated. The first line of the table shows the event-based macro F1-score when using the same parameters as the authors of the respective work. The second line shows the F1-score that was achieved by optimizing the threshold value and median filter size per class (class-dependent) using policy gradient reinforcement learning. Finally, the third line shows the class-independent PG optimization, where only global threshold and median filter window values are optimized and used for all classes. We also test the effect of median filtering on the AED performance by removing it from the post-processing stack to only use thresholding.
For the Baseline AED system, the authors use a fixed event class probability threshold of 0.5 and a fixed window size of 7 for the median filter. Applying median filtering after thresholding improves F1-score by 1.6%, from 38.5% to 40.1%. Class-dependent optimization of thresholding and median filtering parameters via RL leads to a F1-score improvement of 4.4% over the original fixed values, from 40.1% to 44.5%. Class-independent optimization gives a slightly lower improvement of 3.3% over the original values. For the ConformerSED system, the authors use a fixed threshold of 0.5 but perform a coarse grid search separately for each event class to tune the median filter length. Classdependent RL optimization computes on a class basis better values for the median filter length, as the F1-score improves from 47.7% to 52.4%, an increase of 4.7%. Again, classindependent optimization performs slightly worse than the class-dependent one, with an improvement of 3.4% over the original values.
Overall, a reinforcement agent that earns to compute "optimal" values for class thresholds and median filter sizes VOLUME 4, 2016 using policy gradient optimization, yields performance improvements for both AED methods under study. It can also be observed that class-dependent optimization yields better results compared to the class-independent (global) optimization case.
As PG optimizes for a reward-based objective, we can use it to indirectly optimize a non-differentiable objective, such as the F1-score in our case, by translating increases or decreases of that objective to positive or negative credit assignment. Since the F1-score is in the [0%, 100%] range of values, we simply do a straightforward linear mapping to a reward in the [0.0, 1.0] range.

V. LIMITATIONS
The computational time that is required to learn the policy function with a neural network approximator can be nontrivial, depending on the number of audio segments. Given that each audio segment is treated as a state of the MDP, the MDP will grow larger with the amount of audio segments. This in turn will increase the time required for the PG algorithm to reach convergence. In a standard experimental setup with low-cost hardware (Intel Core i9-10900K processor, 32GB RAM) convergence was achieved after approximately 2 hours of processing time on the DCASE-Task4 training dataset, which consists of approximately 2500 audio segments. However, the computational time required is, in our opinion, offset by the superior final classification accuracy achieved by our method. As a final remark, note that our method needs to be trained separately on each AED system output, i.e., we do not provide a global RL agent for all possible AED methods.

VI. CONCLUSION
In this work we introduced reinforcement learning, and in particular policy gradient optimization, as a way of discovering optimal parameter values for the two most commonly used stages of the post-processing stack of an audio event detector: thresholding and median filtering. To that end, we reformulate this parameter optimization problem as a Markov Decision Process, where: 1) the state S consists of the class probability outputs of the AED system for a given audio segment of the dataset, 2) the actions A are the values selected as optimal by the RL algorithm for the thresholding and median filtering parameters, either on a per-class basis or globally, 3) the reward R is the event-based macro F1-score achieved by the RL algorithm over the entire dataset using the selected parameter values for the post-processing stack.
We tested our method on two AED systems submitted to the DCASE Task4 (Sound Event Detection in Domestic Environments) 2020 challenge: 1) A baseline AED system provided by the challenge organizers, 2) The system (Con-formerSED) that achieved top ranking on the challenge, based on achieved event-based macro-F1 score. Both systems use a thresholding + median filtering post-processing stack. We achieved an improvement of about 4% to 5% on average on the event-based macro-F1 score compared to the perfor-mance of these systems with their original parameterization of the post-processing stack.
A drawback of the proposed method is the requirement to train the PG algorithm separately on the outputs of each AED system. The computational requirements highly depend on the size of the dataset of audio segments. Another important prerequisite for the success of the method is the sufficient exploration of the state space to avoid converging on suboptimal post-processing parameter values, generally known as exploration vs. exploitation dilemma in RL literature. However, when properly tuned, it can provide superior final classification performance for an AED system compared to performing exhaustive grid search for finding the optimal parameter values for the post-processing stack, especially on larger datasets. An interesting avenue of future research would be to simultaneously train the PG algorithm on the predictions of several AED systems for a particular dataset and assess its ability to generalize to other AED systems that it was not trained on. Another possibility for future work would be to try other, more sophisticated than Policy Gradient, reinforcement learning algorithms, with our method, such as Proximal Policy Optimization (PPO) [16] and Soft Actor-Critic (SAC) [17].
PETROS GIANNAKOPOULOS is currently a PhD candidate at the Department of Informatics and Telecommunications, National and Kapodistrian University of Athens, Greece. His research is focused on deep reinforcement learning for audio applications. Previously, he earned an MSc in Computer Science and a BSc in Physics. He also has industry experience as a Machine Learning Engineer and as a Data Scientist, working for startups and research institutions.
AGGELOS PIKRAKIS received the Diploma Engineering degree from the University of Patras, in 1993, and the Ph.D. degree in Computer Science from the University of Athens, in 2002. He is currently an Assistant Professor at the Department of Informatics, University of Piraeus, Greece, teaching courses related to machine learning and audio processing. He is the co-inventor of AI/ML patents and he has coauthored two international textbooks in the English language and more than 50 refereed papers in international peer-reviewed scientific journals/conferences. His research interests include audio analysis, with an emphasis on machine learning algorithms, including deep neural networks, hidden Markov models, and Bayesian architectures and sequence alignment methods. His research work has received award recognition and he was a recipient of the 2019 EURASIP Meritorious Service Award.