Layer Pruning With Consensus: A Triple-Win Solution

Layer pruning offers a promising alternative to standard structured pruning, effectively reducing computational costs, latency, and memory footprint. While notable layer-pruning approaches aim to detect unimportant layers for removal, they often rely on single criteria that may not fully capture the complex, underlying properties of layers. We propose a novel approach that combines multiple similarity metrics for neural network internal representation. Our criterion, called Consensus, leverages shape and stochastic metrics, such as adaptations of the Bures and Procrustes distances, to create a single expressive measure of low-importance layers. Our technique delivers a triple-win solution: low accuracy drop, high performance improvement, and increased robustness to adversarial attacks. With up to 78.80% Floating-Point Operations (FLOPs) reduction and performance on par with state-of-the-art methods across different benchmarks, our approach reduces energy consumption and carbon emissions by up to 66.99% and 68.75%, respectively. Additionally, it avoids shortcut learning and improves robustness by up to 4 percentage points under various adversarial attacks. Overall, the Consensus criterion demonstrates its effectiveness in creating robust, efficient, and environmentally friendly pruned models.