Memory Mechanisms in Cross-Modal Computing Advance Performance to 94.41% with Spiking Networks

The pursuit of energy-efficient artificial intelligence drives research into spiking neural networks, but their ability to learn across different senses remains a significant challenge. Effiong Blessing of Saint Louis University, Chiung-Yi Tseng of Luxmuse AI, and Somshubhra Roy of North Carolina State University, along with colleagues, now present the first comprehensive study of how memory mechanisms within these networks perform across visual and auditory information. Their research reveals striking differences in performance depending on the type of memory used, with some systems excelling at visual tasks but struggling with sound, and vice versa. This work establishes that effective memory design must consider the specific sensory input, and demonstrates a 603-fold improvement in energy efficiency compared to conventional neural networks, paving the way for truly adaptable and efficient artificial intelligence systems.

Cross-Modal Memory Mechanisms in Spiking Networks

This research investigates the effectiveness of different memory mechanisms within spiking neural networks (SNNs) when processing multiple sensory inputs, aiming to identify which memory types are best suited for different tasks and to inform the design of more efficient neuromorphic hardware. Key findings demonstrate that different memory mechanisms excel at specific tasks, highlighting that a one-size-fits-all solution is not optimal, and that the best performance is achieved using specialized architectures tailored to each modality, mirroring how the brain processes information with distinct regions dedicated to different senses. The models demonstrate biologically realistic characteristics, such as sparse representations and modality-specific dimensionality, suggesting they capture aspects of how the brain processes information, and offer significant energy efficiency gains compared to traditional deep learning approaches, making them suitable for edge computing applications. Hierarchical Gated Recurrent Networks (HGRN) demonstrate robust cross-modal capability, achieving consistent performance in both parallel and unified configurations.

Cross-Modal Memory Reveals Performance Disparity

This work presents a comprehensive cross-modal evaluation of memory mechanisms in spiking neural networks, revealing significant differences in architectural performance across visual and auditory tasks. Researchers systematically assessed Hopfield networks, Hierarchical Gated Recurrent Networks (HGRN), and supervised contrastive learning (SCL), achieving 97.68% accuracy on visual tasks with Hopfield networks, but only 76.15% on auditory tasks, a significant 21.53 percentage point gap.

This disparity demonstrates that energy-based associative memory, while effective for spatial pattern completion, is constrained when processing purely temporal sequential data. In contrast, SCL achieved more balanced cross-modal performance, reaching 96.72% visual accuracy and 82.16% auditory accuracy, a 14.56 percentage point gap, suggesting that direct engram formation through metric learning offers representational flexibility suitable for diverse temporal structures.

HGRN consistently performed well across both modalities, achieving 97.48% visual and 80.08% auditory accuracy, positioning hierarchical gating as a robust solution for general-purpose neuromorphic processors. Experiments with unified multi-modal training using HGRN achieved 88.78% average accuracy, matching the performance of parallel HGRN processing, with minor reductions in per-modality accuracy.

Quantitative engram analysis confirmed weak cross-modal alignment, validating the parallel architecture design. Furthermore, all architectures maintained over 97% sparsity and achieved a 603x reduction in operations compared to traditional artificial neural networks, demonstrating substantial energy efficiency gains. These findings establish clear design principles for neuromorphic systems, emphasizing the benefits of parallel architectures for modality-specific optimization and the robustness of HGRN for consistent cross-modal performance.

Cross-Modal Memory Performance in Spiking Networks

This research presents the first comprehensive cross-modal evaluation of memory mechanisms within spiking neural networks, revealing significant differences in architectural performance across sensory domains. The team systematically assessed Hopfield networks, Hierarchical Gated Recurrent Networks, and supervised contrastive learning on both visual and auditory tasks, demonstrating that memory mechanisms offer task-specific benefits rather than universal applicability. Notably, Hopfield networks achieved high accuracy on visual processing but underperformed on auditory tasks, while supervised contrastive learning exhibited more balanced cross-modal performance. These findings establish clear design principles for memory-augmented spiking neural networks, showing that parallel architectures maximise performance through modality-specific optimisation and unified models enable efficient deployment.

Quantitative analysis of engrams further supports these conclusions, confirming weak cross-modal alignment and modality-specific representational efficiency. The research demonstrates a substantial 603-fold increase in energy efficiency compared to traditional neural networks, alongside greater than 97% sparsity, positioning memory-augmented spiking neural networks as viable solutions for edge deployment of multi-sensory artificial intelligence systems. The authors acknowledge that performance differences remain between modalities, and future work should explore transfer learning between sensory domains, extend evaluations to additional sensory inputs, and validate these findings on dedicated neuromorphic hardware. Further investigation into sophisticated multi-task learning strategies could also potentially close performance gaps while maintaining deployment efficiency.