RepFC: Universal Structural Reparametrization Block for High Performance, Lightweight Deep Neural Networks

Structural reparameterization (SRP) employs a bipartite network design consisting of two stages: (1) an expanded multi-branch network during training, which is equivalent to (2) a collapsed lightweight network during inference. This approach improves the task metrics of the efficient inference-time network at the expense of longer training times for the expanded network. Recent works have adopted SRP for handcrafting efficient deep neural networks and have proposed universal multi-branch SRP blocks to expand only the convolutional (CONV) layers during training. However, these training schemes expand uniformly every CONV layer in a given deep neural network (DNN), which becomes impractical for modern large-scale DNNs. This work presents a simple and effective technique for structurally reparameterizing dense layers, unlocking additional potential for the technique. We achieve superior accuracy improvements with our proposed Reparametrizable Fully Connected (RepFC) block, by expanding only the dense layers of the classification head in SOTA DNNs, as opposed to expanding all CONV layers in related works. Additionally, we demonstrate how our RepFC block can be leveraged in transformers, where dense layers are employed throughout the network. Transformer architectures Swin-tiny, EfficientViT-B1 and FastViT-T8 show a 0.2, 0.3 and 0.75 p.p. improvement respectively, on the ImageNet-1K dataset. Our experiments demonstrate a 1.46, 0.85 and 1.14 p.p. improvements in AlexNet, MobileNet-V3-L and EfficientNet-B0 respectively on the ImageNet-1K dataset, with zero overhead in inference time. Code is available at https://github.com/shamvbs/RepFC. ¹¹Correspondence to: shambhavi.balamuthu-sampath@tum.de