Adaptive Spiking Neural Networks Enhance Learning and Reduce Energy Consumption.

Research demonstrates improved performance in spiking neural networks (SNNs) through adaptive gradient learning. By dynamically aligning surrogate gradients with membrane potential dynamics, the method mitigates gradient vanishing and enhances solution space exploration, resulting in low-latency, high-performing networks with increased neuronal activation.

Neuromorphic computing, which seeks to emulate the brain’s efficiency, increasingly focuses on spiking neural networks (SNNs) as a potential pathway to low-energy artificial intelligence. A key challenge in training these networks lies in accurately calculating gradients – the signals that guide learning – given the discontinuous nature of spiking activity. Researchers are now addressing this through surrogate gradient learning, approximating these gradients using continuous functions. However, the effectiveness of this approach is limited by the dynamic range of neuronal membrane potentials. Jiaqiang Jiang, Lei Wang, Runhao Jiang, Jing Fan, and Rui Yan, from Zhejiang University of Technology and Zhejiang University, detail their work in “Adaptive Gradient Learning for Spiking Neural Networks by Exploiting Membrane Potential Dynamics”, presenting a method that dynamically aligns gradient estimation with evolving membrane potential distributions, thereby improving learning and reducing the vanishing gradient problem.

Adaptive Gradient Learning Improves Spiking Neural Network Performance

Spiking neural networks (SNNs) offer a biologically plausible computational model with potential advantages in energy efficiency, and are therefore a focus for developing more efficient artificial intelligence systems. Current training methodologies rely on surrogate gradient (SG) learning – a technique approximating gradients of spiking activity using continuous functions – enabling the application of standard deep learning optimisation algorithms to SNNs. However, a limitation exists: the distribution of membrane potential dynamics (MPD) – the fluctuating electrical charge within neurons – often deviates from the fixed range optimally usable by standard SG methods, hindering solution finding and limiting learning capacity.

This research introduces adaptive gradient learning (MPD-AGL), a novel method dynamically aligning the SG with the evolving MPD. This establishes a more flexible and effective training paradigm by fully considering factors influencing membrane potential shifts and creating a dynamic association between the SG and MPD at different time steps. This relaxation of gradient estimation expands the solution space available to the network, enabling discovery of more optimal solutions.

Experiments demonstrate strong performance with low latency across several image datasets, including CIFAR-10, CIFAR-100, CIFAR10-DVS, and Tiny-ImageNet, showcasing the method’s versatility. The researchers evaluated MPD-AGL on various network architectures – ResNet-19, VGGSNN, and VGG-13 – further demonstrating its adaptability and compatibility with different SNN designs. Data augmentation techniques, such as random horizontal flips, rolls, and AutoAugment, were employed to improve model generalisation and robustness. For the event-based CIFAR10-DVS dataset, event streams were segmented into ten time steps.

The study highlights the importance of aligning surrogate gradients with evolving MPD within SNNs, demonstrating that this alignment is crucial for effective learning and improving generalisation to unseen data. By exploiting MPD, the adaptive gradient learning method relaxes gradient estimation and mitigates the gradient vanishing problem, allowing the network to learn more complex patterns by increasing the proportion of neurons operating within the gradient-available interval.

Max-pooling (MP) consistently outperforms spike-timing-dependent plasticity (STDP) as a training method for SNNs, offering a more robust and efficient alternative. The results indicate that MP achieves comparable, and in some instances superior, performance to traditional artificial neural network (ANN) training methodologies, demonstrating its potential as a viable alternative to traditional deep learning approaches. The robustness of MP to variations in hyperparameters simplifies the training process and reduces the need for extensive tuning, easing deployment of SNNs in real-world applications. MP demonstrably requires less training time and computational resources than STDP, offering a significant efficiency advantage.

Future work should investigate the application of MP to larger and more complex datasets and network architectures, expanding the scope of this research and exploring its potential for more challenging tasks. Exploring the potential of combining MP with other advanced learning techniques, such as neuromodulation or attention mechanisms, could further enhance SNN performance and unlock new capabilities, potentially leading to the development of more powerful and efficient SNNs.