An Integer-Only Quantized Transformer with LayerNorm Removal and Linear Attention for Efficient MCU Deployment in TinyML Applications

We propose a structurally optimized Transformer model for efficient deployment on resource-constrained MCUs. Our design removes LayerNorm, Softmax, and GELU, introduces Linear Attention, and adopts I-BERT–style INT-only quantization. When evaluated on an ARM Cortex-M7 (NUCLEO-F746ZG), the model reduced total execution cycles by 24.6%, attention cost by 23.1%, and feedforward operations by 64.1%. Parameter count was also reduced by 3.9% through the removal of LayerNorm and bias terms. These results show that the proposed combination of structural simplification and integer-only quantization enables accurate and efficient inference in TinyML applications on MCU hardware.