SPVT: Spiked Pyramid Vision Transformer

Spiking Neural Networks (SNNs), characterized by spike-based inputs and outputs, exhibit lower power consumption but typically demonstrate modest network performance. On the other hand, the Transformer model excels in image classification, yet its high parameter and computation requirements result in increased energy consumption, posing a significant challenge. This paper introduces a model called Spiked Pyramid Vision Transformer (SPVT), which is structured into four stages. Each stage comprises patch embedding and encoder blocks, with the encoder block integrating spiking self-attention modules. Additionally, IF neurons are employed in SPVT, exhibiting low energy consumption. As the network depth increases, the number of output tokens gradually decreases while the channels progressively increase. Experimental results demonstrate that compared to State-of-the-Art (SOTA), SPVT improves classification accuracy (top 1) from 94.71%/77.35% to 95.53%/78.39% on CIFAR10/100 respectively while using fewer parameters (5.01M vs 9.32M) and consuming less power (238.54uJ vs 255.51uJ).