Neuroscience Institute researchers use AI to model brain’s image sense.

Maggie Henderson’s team at Carnegie Mellon University’s Neuroscience Institute is collecting functional magnetic resonance imaging (fMRI) data to build models predicting how the human visual cortex responds to images. For decades, neuroscience and artificial intelligence have separately pursued the question of how intelligence emerges; now, advances in both fields are enabling a collaborative approach.

Henderson studies how the brain transforms complex perceptual input into meaningful representations of objects and scenes, noting that “for years, that was a difficult challenge.” This interdisciplinary work aims to understand intelligence, both biological and artificial, and open new avenues for innovation in both fields.

AI Models Predict Visual Cortex Responses to Images

Advances in artificial intelligence are now allowing researchers to predict activity within the human visual cortex with increasing accuracy, offering insight into how the brain interprets images. This approach moves beyond simply observing brain activity to proactively forecasting how specific areas will react to different visual inputs. Deep neural networks, initially developed for computer vision tasks, have proven instrumental in bridging this gap; the internal representations generated by these systems closely mirror patterns of activity observed in the human brain when processing visual information.

This reciprocal relationship is central to the emerging field of NeuroAI, where insights from both neuroscience and artificial intelligence mutually reinforce progress. “It’s an incredibly exciting time to be working in neuroscience and cognitive science, as we now have models that go directly from a physical input like a sound or image and compute a predicted neural response,” Feather said.

The scale of this research is bolstered by substantial collaborative initiatives, such as the Simons Collaboration on Ecological Neuroscience (SCENE), a ten-year, $80 million initiative dedicated to developing mathematical theories of how brains transform perception into action. Carnegie Mellon professor Xaq Pitkow participates in SCENE, combining neural recording technologies with computational modeling to test theories about brain function.

Simultaneously, the Machine Intelligence from Cortical Networks (MICrONS) project has yielded detailed reconstructions of neural circuitry, mapping a cubic millimeter of mouse brain tissue containing approximately 200,000 cells and over 500 million connections. These large-scale efforts demonstrate a commitment to understanding the brain at an unprecedented level of detail and providing the data necessary to train and validate increasingly accurate AI models.

Pitkow emphasized the potential of this integrated approach, stating, “By combining these incredibly comprehensive measurements of the brain’s structure and function, understanding some fundamental mechanisms of thought may now be within reach.” This convergence of disciplines is not merely about using AI as a tool for neuroscience; it represents a broader effort to understand intelligence itself, both biological and artificial, and to unlock new possibilities in technology, medicine, and brain health. David Badre, NI’s director, noted that the Neuroscience Institute’s work aligns with federal funding priorities focused on advancing artificial intelligence and neurotechnology, seeking new theoretical frameworks for understanding the human brain.

SCENE & MICrONS Projects Advance Brain Circuit Mapping

This collaborative effort seeks to move beyond simply observing brain activity to creating predictive models of how the brain functions during complex tasks, a shift enabled by increasingly sophisticated tools for both data acquisition and analysis. A central aim of MICrONS was to simultaneously measure anatomical connectivity and neural activity within the same brain tissue, allowing researchers to directly link brain structure to its functional output.

This detailed mapping provides a foundational resource for understanding how information flows within the brain and how different neural circuits contribute to specific behaviors. The scale of SCENE and MICrONS underscores a core tenet of NeuroAI: advanced technologies are now capable of measuring brain activity with unprecedented precision, while computational models are essential for interpreting the resulting data. He presented at the NeuroAI Innovation Domain session, focusing on “The BRAIN NeuroAI Roadmap: Closing the Loop Between Natural and Artificial Intelligence,” and participated in a plenary panel outlining the four pillars of the BRAIN Initiative.

These large-scale projects are not isolated endeavors; they reflect a broader national research priority, according to David Badre, NI’s director. This alignment suggests a growing recognition of the potential for NeuroAI to address both fundamental scientific questions and practical challenges in areas such as healthcare and assistive technologies. The interdisciplinary nature of these collaborations, bringing together neuroscientists, machine learning researchers, and mathematicians, is crucial for fostering innovation and accelerating progress in the field.

“You need domain knowledge in both areas in order to work in neuroengineering. We’re training students to be cross disciplinary with a foundation in both areas,” Badre added, emphasizing the importance of cultivating a new generation of scientists equipped to bridge the gap between neuroscience and artificial intelligence. “This is precisely the brand of interdisciplinary science and innovation that thrives at CMU and the Neuroscience Institute.”

Mathematical Frameworks Connect Perception to Computational Principles

Xaq Pitkow of Carnegie Mellon University’s Neuroscience Institute is developing mathematical frameworks to bridge the gap between how brains process information and the principles underpinning computational systems. SCENE, a ten-year, $80 million initiative, unites neuroscientists and machine learning researchers to create these theoretical connections in real-world environments. Pitkow’s approach centers on understanding how the brain represents information, identifies opportunities for action, and makes decisions despite uncertainty, then formalizing those processes mathematically.

This emphasis on mathematical modeling represents a shift in how researchers approach understanding intelligence, moving beyond simply observing brain activity to defining the underlying rules governing it. This detailed mapping of neural circuitry allows researchers to directly relate brain structure to function, providing crucial data for validating and refining these mathematical models. The ambition extends beyond simply mirroring brain function in machines; it’s about identifying the essential computational principles that enable intelligence, regardless of substrate.

Aran Nayebi, assistant professor in the Machine Learning Department and a faculty member in the Neuroscience Institute, studies animal learning to discern what AI systems still lack. Nayebi asserts, “The brain tells us what behaviors AI has yet to reach to survive in the real world,” and that it “supplies us with concrete engineering targets as to what intelligence is.” His NeuroAgents Lab develops AI systems capable of adapting to changing circumstances, prioritizing learning and flexibility over rigid programming.

This approach acknowledges that true intelligence requires continuous exploration and adjustment, mirroring the way animals learn without constant instruction. This alignment underscores a growing recognition that understanding biological intelligence is key to unlocking the full potential of artificial systems, and vice versa. Michael J. Tarr, the Kavčić-Moura University Professor of Cognitive and Brain Science, believes Carnegie Mellon is uniquely positioned to lead this effort, having established itself as a world leader in the study of intelligence through its combined strengths in psychology, neuroscience, and computer science.

Stimulus-Computable Models Aim to Restore Auditory Processing

Jenelle Feather, assistant professor of psychology at Carnegie Mellon’s Neuroscience Institute, is pioneering the use of models to potentially restore auditory processing in individuals with hearing impairments. These models directly predict neural responses from physical inputs like sounds, offering a new pathway for translational research and personalized hearing solutions. Feather’s work integrates neuroscience, cognitive science, and artificial intelligence to replicate biological systems, ranging from human behavior to measured neural responses, with applications extending to advanced brain-machine interfaces.

The potential impact of these models lies in their ability to simulate various types of hearing impairment and measure resulting changes in brain responses. Feather explains, “Imagine having a high-fidelity model of the healthy human auditory system.” Within this model, researchers could simulate hearing loss and observe how the neural code is affected, ultimately designing more effective algorithms for hearing aids or cochlear implants.

This approach moves beyond generalized solutions, aiming to restore a neural code closer to that of a healthy auditory system.

More like this

Artificial IntelligenceCarnegie Mellon and Pitt historians question if AI ‘thinks’ like us

Artificial IntelligenceSelf-Learning Neurons Boost Machine Learning

ScienceNeuromorphic AI: Dynamical Systems for Learning

ScienceNeuroAI: Biohybrid Intelligence Integrates Hardware