In a significant development that underscores the growing parallels between artificial intelligence (AI) and human cognition, researchers at MIT have uncovered striking similarities between large language models (LLMs) and the human brain’s processing mechanisms. The study, led by Zhaofeng Wu, an electrical engineering and computer science graduate student, reveals that LLMs employ a centralized, generalized approach to process diverse data types, much like the human brain’s semantic hub in the anterior temporal lobe.
The findings, which will be presented at the International Conference on Learning Representations, could pave the way for more efficient and versatile AI models capable of handling a wide array of data types. By understanding how LLMs process information across languages and modalities, scientists can potentially enhance their control over these models and develop improved multilingual AI systems.
The researchers based their study on previous work suggesting that English-centric LLMs use English to reason about various languages. Wu and his collaborators delved deeper into the mechanisms underlying LLM data processing in this expanded investigation. They discovered that an LLM, composed of interconnected layers, processes input text by splitting it into tokens, assigning a representation to each token, and exploring relationships between tokens to generate subsequent words in a sequence.
Non-text inputs like images or audio, these tokens correspond to specific regions of an image or sections of an audio clip. The researchers found that the model’s initial layers process data in its specific language or modality, similar to the modality-specific spokes in the human brain. As the LLM reasons about the data throughout its internal layers, it converts tokens into modality-agnostic representations, mirroring the brain’s semantic hub’s integration of diverse information.
The model assigns similar representations to inputs with similar meanings across various data types, suggesting a shared knowledge base that could boost efficiency in processing vast amounts of data. Furthermore, the researchers found that they could predictably alter the model outputs by intervening in its internal layers using English text, even when those outputs were in other languages.
These insights could potentially be leveraged to encourage LLMs to share as much information as possible across diverse data types while still allowing for language-specific processing mechanisms in specific cases. The researchers also propose that these findings could aid in the development of more effective multilingual models and further studies linking AI models and brain function and cognition in humans.
Mor Geva Pipek, an assistant professor in the School of Computer Science at Tel Aviv University, who was not involved with this work, commended the study, stating, “The hypothesis and experiments nicely tie and extend findings from previous works and could be influential for future research on creating better multimodal models and studying links between them and brain function and cognition in humans.” The research is funded, in part, by the MIT-IBM Watson AI Lab.
The Semantic Hub: A Brain-Inspired Architecture
Neuroscientists posit that the human brain has a “semantic hub” in the anterior temporal lobe, which integrates semantic information from different modalities such as visual data and tactile inputs. This hub is connected to modality-specific “spokes” that route information to the hub. In a striking parallel, LLMs employ a similar mechanism by abstractly processing data from diverse modalities in a central, generalized manner.
For instance, an English-dominant model would rely on English as a central medium to process inputs in Japanese or reason about arithmetic, computer code, etc. Furthermore, researchers have demonstrated that they can intervene in a model’s semantic hub by using text in the model’s dominant language to change its outputs, even when the model is processing data in other languages.
Integrating Diverse Data
The research team based their study on prior work suggesting that English-centric LLMs process data across languages and modalities. To test this hypothesis, they passed pairs of sentences with the same meaning but written in two different languages through the model and measured the similarity of the model’s representations for each sentence. They also conducted experiments where they fed an English-dominant model text in a different language and measured its internal representation’s similarity to English versus the input language.
Consistently, the researchers found that the model’s representations were similar for sentences with similar meanings across various data types. Moreover, the tokens processed in the model’s internal layers were more like English-centric tokens than the input data type.
Leveraging the Semantic Hub
The researchers believe LLMs may learn this semantic hub strategy during training because it is an economical way to process varied data, as much knowledge is shared across languages. They also found that they could predictably change the model outputs by intervening in its internal layers using English text when it was processing other languages.
Scientists could leverage this phenomenon to encourage the model to share as much information as possible across diverse data types, potentially boosting efficiency. However, there may be concepts or knowledge that are not translatable across languages or data types, such as culturally specific knowledge, for which LLMs might need language-specific processing mechanisms.
Implications and Future Directions
This research offers valuable insights into how language models process inputs across languages and modalities, a key question in artificial intelligence. The findings could be instrumental in creating better multimodal models and studying links between them and brain function and cognition in humans.
Understanding the semantic hub of LLMs could help researchers prevent language interference in multilingual models, improving their accuracy across languages. Furthermore, exploring how to maximally share information while allowing for language-specific processing mechanisms is an exciting avenue for future work on model architectures.
External Link: Click Here For More
