Flow of Information in Feed-Forward Denoising Neural Networks

Due to inaccuracies in data acquisition, time series data often suffer from noise and instability which leads to inaccurate data mining results. The ability to handle noisy time series data is thus critical in many data-driven real-time applications. Using the locality feature of time series, feed-forward deep neural networks has been effectively used for time series denoising. In this paper, in order to understand the underling behavior of denoising neural networks, we use an information theoretic approach to study the flow of information and to determine how the entropy of information changes between consecutive layers. We develop analytical bounds for multi-layer feed-forward deep neural networks deployed in time series denoising. Numerical experiments support our theoretical conclusions.