The Qualitative Reasoning Group at Northwestern University is integrating large language models (LLMs) into the Companion cognitive architecture, a system designed to create humanlike software social organisms. The team has successfully used BERT, an LLM, to improve the precision of extracted facts from 45.7% to 71.4%. Despite challenges such as sourcing high-quality semantic annotations, the team is exploring further ways to leverage LLMs, including few-shot and zero-shot learning capabilities. The integration of LLMs could significantly enhance the system’s ability to understand and converse in natural language.
The Companion Cognitive Architecture and Large Language Models
The Companion cognitive architecture, developed by the Qualitative Reasoning Group at Northwestern University, aims to create humanlike software social organisms. The architecture emphasizes natural language capabilities for reading and conversation. Recently, the team, including Constantine Nakos and Kenneth D Forbus, has begun experimenting with large language models (LLMs) as a component in the Companion architecture.
Case Study: Using BERT with Companion Architecture
The team has been using BERT, a large language model, with their symbolic natural language understanding system. The integration of BERT has been beneficial in learning by reading, a process explored in Companions in several ways. The team has used BERT to assist with disambiguation and to predict fact plausibility. The integration of BERT has improved the estimated precision of extracted facts from 45.7% (analogy alone) to 71.4% (analogy + BERT).
Advantages and Limitations of CNLU
The Companion cognitive architecture uses CNLU, a rule-based semantic parser that produces interpretations in the CycL knowledge representation language. One of the advantages of CNLU is that its representations are discrete and inspectable. Unlike neural models, errors it produces can be examined and corrected. However, CNLU has its limitations. Disambiguation is a perennial challenge, and lexical and grammatical coverage remains an issue due to the sheer breadth of natural language.
Future Prospects for Using LLMs
The team is considering three further approaches for integrating LLMs into the Companion architecture. They are exploring the use of the few-shot capabilities of LLMs to help CNLU choose between candidate interpretations. The predictive power of modern LLMs allows them to learn new tasks on the fly, reducing the amount of task-specific training data to a few examples (few-shot learning) or none (zero-shot learning).
Challenges in Integrating LLMs
While the integration of LLMs into the Companion architecture has shown promising results, there are challenges. As the coverage of CNLU grows, the coverage of the classifier will need to grow with it. Sourcing high-quality semantic annotations to support a constantly growing language system is a daunting prospect. The team is exploring several schemes for probing the LLM to overcome these challenges.
Conclusion
The integration of large language models into the Companion cognitive architecture has the potential to significantly improve the system’s ability to understand and converse in natural language. While there are challenges to overcome, the initial results are promising, and the team at Northwestern University is actively exploring further ways to leverage the power of LLMs.
“Using Large Language Models in the Companion Cognitive Architecture: A Case Study and Future Prospects” is an article authored by Constantine Nakos and Kenneth D. Forbus. Published on January 22, 2024, in the Proceedings of the AAAI Symposium Series, the paper explores the application of large language models in the Companion Cognitive Architecture. The authors present a case study and discuss future prospects in this field. https://doi.org/10.1609/aaaiss.v2i1.27700
