Soon Filter: Advancing Tiny Neural Architectures for High Throughput Edge Inference

As Deep Neural Networks become more complex and computationally demanding, efficient models for inference at the edge, particularly multiplication-free ones, have gained significant attention. The Ultra Low-Energy Edge Neural Network (ULEEN) is a notable architecture optimized for high throughput edge designs. ULEEN uniquely employs Bloom Filters with binary values to compute neuron activation, boasting better efficiency metrics than Binary Neural Networks (BNNs). This work uncovers a gradient back-propagation bottleneck within ULEEN’s Bloom filters and introduces a simplified version of it as a solution: the "Soon Filter". Both theoretically and empirically, we demonstrate that our approach improves gradient back-propagation efficiency. Tests on MLPerf Tiny, MNIST and various UCI datasets reveal that our method surpasses ULEEN, BNN, and DeepShift. Notably, with MLPerf KWS (Key Word Spotting) dataset, we achieve 69.6% accuracy with only 101KiB, while ULEEN, BNN and DeepShift achieve only 67.4%, 55.9%, and 24.9% respectively. Remarkably, we also achieve 67.7% accuracy with only 50KiB, resulting in a 2x model size reduction compared to ULEEN while maintaining similar accuracy (+0.3%). This results underscores the promising potential of our solution for efficient inference at the edge in applications that rely on high throughput architectures.