Loading [MathJax]/extensions/MathMenu.js
On Cross-Layer Alignment for Model Fusion of Heterogeneous Neural Networks | IEEE Conference Publication | IEEE Xplore

On Cross-Layer Alignment for Model Fusion of Heterogeneous Neural Networks


Abstract:

OTFusion, or layer-wise model fusion via optimal transport, applies soft neuron association to unify different pre-trained networks. Despite its effectiveness in saving c...Show More

Abstract:

OTFusion, or layer-wise model fusion via optimal transport, applies soft neuron association to unify different pre-trained networks. Despite its effectiveness in saving computational resources, OTFusion requires the input networks to have the same number of layers. To address this issue, we propose a novel model fusion framework, named CLAFusion, to fuse neural networks with different numbers of layers, which we refer to as heterogeneous neural networks, via cross-layer alignment. We demonstrate that the cross-layer alignment problem, which is an unbalanced assignment problem, can be solved efficiently using dynamic programming. Based on the cross-layer alignment, our framework balances the number of layers of neural networks before applying layer-wise model fusion. Our experiments indicate that CLAFusion, with an extra finetuning process, improves the accuracy of residual networks on the CIFAR10, CIFAR100, and Tiny-ImageNet datasets. Furthermore, we explore its practical usage for model compression and knowledge distillation when applied to the teacher-student setting.
Date of Conference: 04-10 June 2023
Date Added to IEEE Xplore: 05 May 2023
ISBN Information:

ISSN Information:

Conference Location: Rhodes Island, Greece

References

References is not available for this document.