Advances in AI, SymSeqBench Benchmarks Sequence Learning for Cognitively Relevant Applications

Sequential structure underpins many aspects of natural intelligence, from the intricacies of language to the patterns of movement and decision-making, and replicating this ability remains a key challenge for artificial intelligence. To address this need, Barna Zajzon from the Institute for Advanced Simulation (IAS-6), Jülich, alongside Younes Bouhadjar, Maxime Fabre, and colleagues, present SymSeqBench, a novel framework for both creating and analysing rule-based symbolic sequences. This unified system combines two tools, SymSeq and SeqBench, to rigorously generate structured sequences and then evaluate how well artificial learning systems process them, offering a versatile platform applicable to fields ranging from psycholinguistics to cognitive psychology and artificial intelligence. By grounding its approach in Formal Language Theory, SymSeqBench provides researchers with a standardised method for conceptualising experiments and, crucially, facilitates a shared understanding of cognition and behaviour through formalised frameworks.

Research Team And Author Contributions

A comprehensive list of contributors to this research includes Gabriel Achiam, Soikat Hasan Ahmed, Sidharth Annapragada, Andreas G. Andreou, Chiara Bartolozzi, Arindam Basu, Petrut Bogdan, Sonia Buckley, Dean V. Buonomano, Michael Furlong, William Gilpin, Aditya Gilra, Hector A. Gonzalez, Giacomo Indiveri, Siddharth Joshi, Vedant Karia, Lyes Khacef, James C. Knight, Muir Petrovici, Christoph Posch, Alpha Renner, Yulia Sandamirskaya, Clemens J.

Schaefer, Stewart, Jonathan Timcheck, Nergis Tömen, Gianvito Urgese, Marian Verhelst, Craig M. Vineyard, Bernhard Vogginger, Amirreza Yousefzadeh, Fatima Tuz Zohora, Charlotte Frenkel, Vijay Janapa Reddi, Benjamin Wilson, Felix Hao Wang, Toben H. Mintz, Terry A. Welch, Pearl, Libby Zhang, Red Hoffmann, Sofia Makowska, Winthrop F. Gillis, Maya Jay, Alexander Mathis, Mackenzie W.

Generating and Evaluating Symbolic Sequence Complexity

The research team developed SymSeq and SeqBench, complementary software tools for generating and analyzing structured symbolic sequences and evaluating artificial learning systems. SymSeq generates sequences using regular grammars with an indexed, Markov-style notation, allowing for unambiguous state distinction and Markov analysis. Researchers can create these grammars by inferring them from existing sequences, loading presets, manually specifying parameters, or, uniquely, randomly generating them with prescribed complexity, enabling systematic evaluation of computational constraints. This random generation allows controlled investigation of how sequence complexity impacts both biological and artificial models.

The study pioneered a hierarchical organization of metrics to analyze sequential data, ranging from individual symbols to complete generative grammars, reflecting the nested organization of sequential structure. Token-level metrics quantify basic statistical properties like frequency and temporal persistence, while string-level metrics evaluate internal organization using measures such as Shannon entropy and Lempel-Ziv complexity. String-set level metrics characterize relationships across sequences, employing edit distance, mutual information, and psycholinguistic metrics assessing predictability. The most innovative aspect is the grammar-level analysis, where metrics characterize the underlying generative mechanisms, independent of sampling biases, including topological entropy quantifying generative capacity and Markov order estimation providing a language-theoretic characterization.

SymSeqBench Measures Sequence Learning Complexity and Performance

Scientists have developed SymSeqBench, a framework for evaluating sequence learning in artificial intelligence systems, and demonstrated its capabilities through rigorous analysis and benchmark creation. Experiments reveal that SymSeq accurately generates regular grammars of prescribed complexity, enabling systematic control over experimental paradigms and the construction of graded benchmark suites for structure-learning studies. Researchers measured topological entropy (TE) to quantify grammar complexity, finding it the most sensitive and consistent measure when correlating with ambiguity within grammars. Results demonstrate that TE effectively captures combinatorial growth in valid strings, remaining insensitive to symbol identities or transition probabilities, unlike other complexity measures.

Data shows that compressibility emphasizes ambiguous states, whereas Lempel-Ziv-Welch (LZW) fails to accurately reflect increasing complexity with ambiguity depth. Effective Measure Complexity (EMC) displays a bell-shaped dependence on repetition depth, limiting its ability to capture subtle contextual variations. The team also established that SeqBench offers a versatile pipeline for transforming symbolic sequences into task-ready datasets, with fine-grained control over structural complexity, input representations, and symbol semantics, facilitating both theoretical analyses and cognitive experiments.

SymSeqBench, A Framework for Sequential Analysis

This research presents SymSeq and SeqBench, tools designed to rigorously generate and analyze structured symbolic sequences and evaluate artificial learning systems. By combining these resources as SymSeqBench, the team offers a versatile framework for investigating sequential structure across diverse fields including psycholinguistics, cognitive psychology, behavioural analysis, and artificial intelligence. The team demonstrated the utility of SymSeqBench by applying it to analyse sequences from the behaviours of several animal species, revealing differences in the complexity and predictability of their actions.

Data suggest that zebrafish and finch exhibit more complex and flexible behaviours compared to mouse, seal, and turtle, as indicated by measures of sequence entropy and string-set complexity. While acknowledging the challenges of inferring underlying generative grammars from observed sequences, the researchers employed a pragmatic approach, utilising a combination of statistical metrics to characterise sequence complexity and dependencies. The authors note that determining the precise generative grammar from behavioural data is computationally intractable, and their methodology prioritises feasibility. Future work could focus on developing more robust methods for handling limited data and refining the statistical metrics used to characterise sequence structure, potentially incorporating negative examples to improve grammatical inference. Nevertheless, SymSeqBench provides a valuable, openly available resource for researchers seeking to investigate sequential behaviour and advance our understanding of the cognitive processes that underpin it.