Quantization-Aware Training for Autoencoder-Based Partitioning of CNNs

We enhance Convolutional Neural Network (CNN) deployment through optimized model partitioning for efficient implementation on resource-constrained IoT devices. Our strategy involves (a) a novel trainable activation sharing function and (b) combining lossy and lossless compression at the partitioning point. Using an autoencoder, trained via Knowledge Distillation, with activation sharing and LZMA compression, we maximize intermediate feature map compression and accelerate IoT-Server communication. Experiments on ResNet50 and YOLOv10n, implemented on a Raspberry Pi 4 with LTE-C4 and WiFi(ah), demonstrate up to 7× speed-up compared to full in-node processing, with less than 1% accuracy drop and 99% IoT-model size reduction via 3-bit quantization (ResNet50 on Tiny-ImageNet-200). We analyze latency across key processes, highlighting significant improvements in IoT-node latency and performance preservation. Our approach enhances existing methods in model partitioning and quantization, offering solutions for reducing latency on the IoT-node while maintaining performance.