A typical large language model will deliver fifty disconnected answers when presented with fifty related documents, treating each upload as a fresh start with no memory of previous information. Andrej Karpathy describes this limitation as a break from a typical loop, envisioning an AI that instead maintains a structured, cross-linked knowledge base.

Karpathy explains, “LLMs don’t get bored, don’t forget to update a cross-reference, and can touch 15 files in one pass.” To test this “wiki” approach, one researcher specifically stress-tested the system using The Hound of the Baskervilles, evaluating its ability to update knowledge as new plot details emerged.

LLM Wiki Design: Separating Sources, Wiki, and Schema

Maintaining three distinct layers, raw source documents, a dynamically updated wiki, and a governing schema, is a deliberate architectural choice of the system, and it is important for persistent knowledge retention. Unlike typical large language model interactions where each uploaded document is treated as a fresh start, this design prioritizes building a continuously refined knowledge base. The raw sources remain untouched, serving only as input; the wiki comprises markdown pages authored and maintained by the AI itself, evolving with each new document processed.

This wiki layer expands with summaries, entity pages, and conceptual comparisons, becoming increasingly detailed over time. The schema functions as a set of rules defining the structure and metadata of the wiki, transforming the AI from a simple chatbot into a disciplined knowledge manager.

Ingestion is the core operation, where a new source triggers updates or creations across multiple wiki pages, potentially touching ten to fifteen pages with a single document. The entire process is initiated with a simple command, such as typing /llm-wiki-ingest chapter-08-first-report-of-dr-watson, which directs the AI to consult the schema, identify relevant pages, and then index the new information. This emphasis on disciplined bookkeeping, rather than complex intelligence, is central to the system’s effectiveness.

A key test involved The Hound of the Baskervilles, evaluating the wiki’s ability to revise its understanding as plot twists unfolded; after the first chapter, the wiki had exactly what you’d expect: stub pages for characters like Holmes and Watson, and locations like Baskerville Hall. Each wiki page adheres to a standardized format, including required fields like type, title, and last-updated date.

Page types are deliberately limited to a controlled vocabulary: character, location, artifact, event, and concept were chosen for the Hound project because of the novel’s structure. The addition of new types is considered if existing types are inadequate. This rigid structure enables the build and lint tools to validate and index new pages without requiring custom code, as the schema itself remains agnostic to the specific content.

“A schema anyone can adopt is a wiki anyone else’s tooling can read,” explains the creator, highlighting the potential for interoperability and shared knowledge bases. This approach is best suited for bounded, curated datasets, a few hundred documents rather than millions, where valuable insights require connecting information from multiple sources. The system excels in scenarios like research projects, book studies, or personal note-taking over extended periods.

While a retrieval layer may become necessary with larger wikis, around two hundred pages, the design allows for a hybrid approach, combining the benefits of both wiki-style knowledge organization and traditional information retrieval. The current design assumes a single author or agent, lacking access control or a transaction model; concurrent editing is considered a conflict, not a supported feature.

A flawed inference, if left unchecked, can become permanently embedded within the wiki. “Major: Vandeleur alias added; St. Oliver’s school named; portrait = Hugo’s face; dictated the L.L. letter; Laura Lyons mechanism exposed,” details a log entry documenting the evolving understanding of the novel’s complex web of characters and events.

LLMs don’t get bored, don’t forget to update a cross-reference, and can touch 15 files in one pass.

Hound of the Baskervilles: Testing Wiki Updates with a Plot Twist

Demonstrating a capability to reconcile conflicting information, the system was immediately tested by discrepancies within Arthur Conan Doyle’s The Hound of the Baskervilles. Subsequent chapters presented evidence contradicting the initial assessment of Barrymore as a key suspect. This forced the system to update its internal “wiki” pages, demonstrating a capacity for revision not typically found in large language models. “Chapter 7 says Barrymore. Chapter 13 says otherwise,” noted the development logs, detailing the evolving understanding of the novel’s characters.

The core of this approach lies in separating raw source documents from the structured wiki itself, and a schema defining the organization of information. This streamlined interface highlights that the intelligence isn’t in the reading, but in the meticulous bookkeeping.

The creator restricted the vocabulary of page types to character, location, artifact, event, and concept, tailoring the system to the demands of a novel. This constraint is not arbitrary; it reflects the intended use case for this type of system. A research project, a detailed book study, or a year’s worth of personal notes are ideal applications. The Hound of the Baskervilles test also revealed nuanced inferences made by the system.

A log entry detailed, “Author attribution revised: Lyons wrote it but Stapleton dictated it,” demonstrating the AI’s ability to deduce complex relationships between characters and actions. Another entry, “Telegram alibi not confirmed… Watson now holds open the possibility he was the London spy,” showed the system’s capacity to evaluate evidence and revise its assessment of a character’s motives.

These updates weren’t simply factual corrections; they reflected a deeper understanding of the narrative’s intricacies. The creator emphasizes that this approach is best suited for situations where connecting information across multiple sources is important, rather than simply looking up a single fact.

Major: Vandeleur alias added; St. Oliver’s school named; portrait = Hugo’s face; dictated the L.L. letter; Laura Lyons mechanism exposed.

Wiki Ingest Process: Updating Pages with Evidence & Hypotheses

Relying on deterministic tools to ensure reliability, the system’s ingest process moves beyond simple prompting. A command-line interface, wiki_tool. py, facilitates core operations like building a table of contents and validating page metadata, establishing a foundation for consistent knowledge maintenance. This tool executes specific tasks: regenerating the wiki’s table of contents, flagging orphaned pages, and providing a ranked-retrieval layer for efficient querying.

A typical run following an ingest session produces output like this: “Built – 23 concept(s), index.md and catalog.jsonl updated,” alongside reports identifying orphaned pages such as “Wiki/Locations/northumberland-hotel.md: orphan – no inbound [[wiki-links]] from other notes.” The design prioritizes catching “drift,” or the accumulation of inconsistencies, through these automated checks. The tools are invoked directly, ensuring that the wiki remains coherent and accurate as new information is incorporated.

This contrasts sharply with retrieval-augmented generation (RAG) systems, which reconcile contradictory information from source documents with each query, effectively repeating the work with every request. Here, reconciliation occurs once during ingest, allowing subsequent queries to access already-corrected answers. The architecture acknowledges a trade-off between upfront cost and ongoing query expense. RAG systems incur a small, flat cost for each query but do not retain synthesized knowledge, making them suitable for rapidly changing corpora.

Conversely, a wiki demands an initial investment in reconciliation during ingest, but subsequent queries benefit from the already-corrected information. The creator notes that a retrieval layer becomes necessary around two hundred pages, at which point the system transitions to a hybrid approach. This scaling point doesn’t represent a limitation, but rather an opportunity to combine the strengths of both methods.

Concurrent editing is treated as a conflict, a deliberate choice reflecting the system’s focus on maintaining a consistent, authoritative record. While a wrong inference can become settled within the wiki, deterministic tools, the lint function, are implemented to identify and correct errors.

Telegram alibi not confirmed… Watson now holds open the possibility he was the London spy.

Deterministic Bookkeeping: Building Trust Beyond LLM Prompting

Maintaining a consistent understanding across multiple documents presents a distinct challenge for large language models, as repeated analysis of the same information is required with each new query. Unlike traditional retrieval-augmented generation (RAG) systems, this approach prioritizes persistent knowledge retention through a structured “wiki” format, where the AI actively writes and maintains summaries, entity pages, and concept comparisons.

The system, tested extensively with The Hound of the Baskervilles, demonstrated an ability to reconcile contradictory information, such as differing accounts of Barrymore’s actions in chapters seven and thirteen, during the initial ingest process, rather than re-evaluating the discrepancy with each subsequent question. Asking the wiki who the suspects are after processing the novel yields a clean answer identifying Barrymore, Laura Lyons, and Stapleton, while a RAG system would repeatedly reconcile conflicting details from various chapters.

The interface for adding new information is remarkably simple; typing triggers the system to read the schema, determine relevant wiki pages, and validate the resulting index. Several unexpected considerations emerged during development. The system’s scalability is limited by the size of the index that must fit within a language model’s context window, necessitating a retrieval layer beyond a certain point. While the wiki can inadvertently settle on incorrect inferences, deterministic tools, the lint function, are implemented to identify and correct errors.