Accurate data identification in low signal-to-noise ratio series

In this paper we will present our unsupervised machine-learning-based method to filter signals with a low signalto-noise ratio. We will describe both the generalized version of the algorithm and a simplified example using a discrete, bidimensional signal. First, we will explain what unsupervised machine learning is and how a mathematical model can be trained to differentiate the useful signal from the noise. We will describe how we represent our data in a feature space, by extracting measurable properties of the signal and using them as features. Afterwards, we use a machine learning algorithm to group the signal into two different clusters, separating the noise from the true signal based on the aggregated differences between their features. The model is trained on data that is updated continuously and in real-time, with the need of minimal information beforehand. Secondly, we will detail the pipeline of our algorithm. The main focus of our work was to extract as many relevant features as possible from the signal. For this, we associated signal samples in spatially correlated groups, according to the Euclidean distance between them. Each group of signal samples forms a cloud of points, with each sample being one point. Furthermore, using the Minimum Volume Enclosing Ellipsoid algorithm, we construct an ellipsoid for each cloud of points. This was done in order to extract more useful information about the signal, based on its spatial distribution. Having that ellipsoid, we were able to use its geometrical properties as new features, such as its axis length, ratio and tilt angle. The next step in the filtering process was to concatenate all these features into a descriptor. A data-set was formed from the last n time steps, based on which the k-means algorithm clusters all the clouds in two different groups, based on their descriptors. Finally, in order to fairly evaluate the algorithm, we created a testing environment: a synthetic signal will be generated following basic patterns with random noise added. This will help by providing some useful metrics of accuracy in order to evaluate the proposed method.