DCKD: Bridging DETR and CNN-based Detectors with Decomposed Knowledge Distillation

Object detection has significantly progressed with Detection Transformer (DETR) models, which achieve high accuracy through Transformer architectures. However, DETR’s substantial computational demands limit its suitability for real-time applications on resource-constrained devices, where lightweight Convolutional Neural Networks (CNNs) are preferred. While knowledge distillation (KD) is an appealing technique to compress giant detectors into smaller ones, existing KD methods struggle to efficiently transfer knowledge from DETRs to CNN-based detectors due to a substantial semantic gap. To address this, we propose the DETR-CNN Knowledge Distillation (DCKD) framework. Our framework decomposes the knowledge in DETR into Homogeneous and Heterogeneous categories and introduces a novel approach integrating Heterogeneous Knowledge Logits Distillation (HeKLD) and Homogeneous Knowledge Feature Distillation (HoKFD) to bridge the semantic gap between DETR and CNN-based detectors. HeKLD aligns cross-head prediction outputs to bridge architectural differences and ensure consistent decision-making, while HoKFD transfers low-level features by focusing on relevant regions to enhance detection accuracy. Extensive experiments on the COCO dataset demonstrate that our DCKD significantly improves mean Average Precision (mAP) across various teacher-student pairs, consistently outperforming the latest state-of-the-art KD methods.