Successive Information Bottleneck and Applications in Deep Learning

Information Bottleneck (IB) method studies the trade-off between compression and prediction: extracting relevant information from the input variable X while preserving relevant information about another random variable Y , the resulting representation is another random variable Z. Motivated by the deep neural networks implementations, this paper studies a novel variation of the N-layer IB problem where the layers are assumed to be encoded in a successive fashion. We propose a method for performing an N-Layer IB in a greedy fashion and analyze numerical results obtained over a set of synthetic experiments.