Scaling Equilibrium Propagation to Deeper Neural Networks Surpasses Backpropagation in Accuracy

Equilibrium propagation offers a compelling alternative to the standard backpropagation algorithm, potentially paving the way for more biologically realistic and efficient neural networks. Sankar Vinayak E. P. and Gopalakrishnan Srinivasan, from the Indian Institute of Technology, Madras, and their colleagues, now demonstrate a significant advance in this field by successfully scaling equilibrium propagation to much deeper network architectures. Previous attempts at equilibrium propagation struggled to achieve high accuracy or were limited to relatively shallow networks, leaving a considerable performance gap compared to backpropagation. This team overcomes these limitations with the introduction of the Hopfield-Resnet architecture, which incorporates residual connections and a clipped activation function, enabling the training of networks with nearly twice the number of layers previously achieved. The resulting Hopfield-Resnet13 network attains 93. 92% accuracy on the CIFAR-10 dataset, a 3. 5% improvement over prior results and comparable to the performance of a similarly sized network trained with backpropagation.

Residual Connections Stabilize Deep Hopfield Networks

This research introduces improvements to deep convolutional Hopfield networks, specifically enabling them to scale to greater depths and achieve performance comparable to traditional feedforward networks trained with backpropagation. The authors address the challenges of training deep Hopfield networks using Equilibrium Propagation (EP), an alternative learning algorithm. Key contributions include the introduction of residual connections between hidden layers, allowing EP to train deeper networks effectively, and the utilization of ReLUα as a non-linear activation function, improving training stability. The proposed architecture achieves significant accuracy gains on image datasets (CIFAR-10, CIFAR-100, Fashion MNIST), surpassing previous results with Hopfield networks.

EP-trained networks exhibit distinct weight characteristics compared to backpropagation-trained networks, with smaller absolute weight values, lower weight variance, and a tendency for weights in deeper layers to approach zero. Residual connections help mitigate this sparsity in deeper layers. The research demonstrates that by incorporating residual connections and a specific activation function, deep convolutional Hopfield networks can be effectively trained with Equilibrium Propagation, offering a viable alternative to backpropagation for certain applications. Key concepts include Hopfield Networks, Equilibrium Propagation, Residual Connections, and Energy-Based Models.

Deeper Networks Trained with Hopfield-Resnet Architecture

Scientists have achieved a significant breakthrough in neural network training with the development of the Hopfield-Resnet architecture, offering a biologically plausible alternative to traditional backpropagation. This work addresses limitations in prior equilibrium propagation methods, which were restricted to shallower networks. The team successfully trained networks with nearly twice the number of layers previously reported, demonstrating a pathway towards deeper, more complex artificial intelligence systems. Experiments on the CIFAR-10 dataset revealed that the Hopfield-Resnet13 architecture achieved 93.

92% accuracy, a 3. 5% improvement over the previous best result and comparable to the performance of a Resnet13 network trained using backpropagation. On the CIFAR-100 dataset, the architecture attained 71. 05% accuracy, and on Fashion-MNIST, it reached 94. 15%.

Analysis revealed that utilizing the ReLU6 activation function, combined with enhanced data augmentation and residual connections, yielded higher accuracy than the ReLU1 function. These modifications enabled deeper networks to outperform shallower ones, demonstrating the viability of this alternative training paradigm. Memory utilization during training was comparable to that of a similarly sized feedforward network. These results demonstrate a pathway towards more efficient and biologically inspired artificial intelligence systems.

Residual Connections Scale Equilibrium Propagation Deeply

This work introduces the Hopfield-Resnet architecture, a significant advancement in biologically plausible learning algorithms. By incorporating residual connections into convolutional Hopfield networks, researchers have successfully scaled equilibrium propagation (EP) to deeper networks than previously possible. The resulting Hopfield-Resnet13 achieves 93. 92% accuracy on the CIFAR-10 dataset, a 3. 5% improvement over prior results and performance comparable to that of a similarly sized network trained with traditional backpropagation.

These improvements demonstrate the potential of EP as an alternative to backpropagation for training deep neural networks. The team observed that the use of residual connections reduces the prevalence of near-zero weight values during training, mitigating challenges associated with training deeper networks. This architectural enhancement, combined with the adoption of ReLUα as a non-linear activation function, yielded substantial accuracy gains across multiple datasets. Future research directions include integrating these architectures with approaches for sequence learning and developing specialized hardware and algorithmic optimizations to improve computational efficiency and fully realize the potential of EP.