Explaining Deep and ResNet Architecture Choices with Information Flow

Recently, Information Theoretic Learning (ITL) has helped explain the learning dynamics for deep learning models such as multilayer perceptrons (MLP), convolutional neural networks (CNN), and stacked autoencoders (SAE). It is understood that for MLPs and CNNs, where the desired signal and input are independent of each other, the set of consecutive layers in the primary and adjoint networks represent individual Markov Chains (MC) that effect two data processing inequalities (DPI) in their respective directions. For the SAE, the desired signal is the input, so the DPI is only confirmed until the bottleneck layer. In this paper, we propose using the adjoint network to compute conditional mutual information with the backpropagated errors to demonstrate the DPI until the SAE's last layer. Also, we present an ITL-based analysis of the residual network (ResNet) architecture and propose an explanation for why the identity mapping is the optimal shortcut connection.