Spiking Neural Networks’ Limited Receptive Fields Constrain Visual Long-Sequence Modeling Capabilities

Spiking Neural Networks (SNNs) offer a promising pathway to energy-efficient computing, yet currently struggle to match the performance of conventional networks when processing extended visual information. Jieyuan Zhang, Xiaolong Zhou, Shuai Wang, and colleagues address this challenge by introducing a new method for understanding how SNNs process visual data over time and space. The team developed the Spatio-Temporal Effective Receptive Field (ST-ERF) framework, which reveals that existing SNN models often fail to establish a comprehensive understanding of the entire visual scene across extended sequences. To rectify this limitation, the researchers propose two innovative architectural components, MLPixer and SRB, that significantly enhance the network’s ability to integrate information across both space and time, leading to substantial improvements in object detection and semantic segmentation tasks. This work not only advances the performance of SNNs, but also provides a valuable analytical tool for designing and optimising future generations of these energy-efficient networks.

Recognizing that SNNs often lag behind traditional artificial neural networks in visual long-sequence modeling, the team investigated how effectively these networks model spatial and temporal dependencies. Inspired by receptive field analysis in neuroscience and artificial intelligence, researchers extended the concept to account for the unique dynamics of spiking neurons, creating a method to quantify how input features contribute to output features through gradient analysis. The study revealed that existing Transformer-based SNNs often fail to establish a robust global ST-ERF, limiting their ability to model long-range spatial dependencies essential for accurate visual processing.

To address this limitation, scientists developed two novel channel-mixer architectures: a multi-layer-perceptron-based mixer (MLPixer) and a splash-and-reconstruct block (SRB). These architectures specifically enhance global spatial ERF throughout all timesteps in the early stages of the network, improving performance on challenging visual tasks. Extensive experiments validated the effectiveness of the proposed method, utilizing Meta-SDT variants and conducting evaluations across object detection and semantic segmentation tasks. Researchers meticulously measured the ST-ERF distributions across various SNN architectures, demonstrating how the MLPixer and SRB architectures expanded the effective receptive field and improved feature modeling. The team’s analysis reveals that existing Transformer-based SNNs struggle to establish a robust global ST-ERF, limiting their ability to model visual features effectively across extended sequences. To address this limitation, researchers proposed two novel channel-mixer architectures: MLPixer and Splash-and-Reconstruct Block (SRB).

These designs enhance the global spatial ERF throughout all timesteps in the early stages of Transformer-based SNNs, improving performance on challenging visual long-sequence modeling tasks. Experiments demonstrate that a Meta-SDT-Base network incorporating the SRB architecture achieves high performance on object detection and semantic segmentation benchmarks, surpassing state-of-the-art Transformer-based SNNs while maintaining a smaller model size. This research establishes a new understanding of SNN bottlenecks and offers a pathway to improved performance in visual long-sequence modeling. The team identified that existing Transformer-based SNNs struggle to establish a robust global spatio-temporal effective receptive field, hindering their ability to effectively model visual features. To overcome this, MLPixer and SRB were designed to enhance global spatial effective receptive fields, particularly in the early stages of network processing. Extensive testing on benchmark datasets, including COCO and ADE20K, demonstrates that both MLPixer and SRB consistently outperform baseline SNN models across object detection and semantic segmentation tasks.

Notably, the SRB variant achieved improvements on object detection metrics and semantic segmentation mean Intersection over Union (mIoU), while also reducing model parameters in some configurations. Further validation on event-based tracking datasets confirms the effectiveness of these architectures in challenging, real-world applications. The authors acknowledge that the performance gains are dependent on the specific network configuration and training schedule employed. Future research directions include exploring the application of these channel-mixer architectures to a broader range of SNN tasks and investigating methods to further optimize their performance and efficiency. The team believes the proposed spatio-temporal effective receptive field framework offers valuable insights for designing and optimizing SNN architectures more generally.