Relationships between Deep Learning and Linear Adaptive Systems

Linear adaptive systems are a well-known staple in numerous signal processing applications. Recently, significant activity and performance gains have been achieved in multilayer neural networks for deep learning applied to practical data processing applications. In this paper, we describe the important relationships and significant differences between the procedures and methods used in linear adaptive systems and those used in multilayer neural networks for deep learning tasks. Input-output structures, cost functions and training criteria, adaptive algorithms, and data processing and optimization strategies are considered. It is the hope of the authors that this discussion will spur further crossover between the two fields, and in particular allow knowledge to be shared and further progress to be made.