For a while now I’ve been trying to build a proper memory layer on top of my research, writing, and content creation. Today it all lives in my Second Brain in Obsidian, where the primitives are files like notes, videos, and articles.

What I actually want is to shift those primitives from files to entities and relationships, such as people, locations, objects, topics, preferences, and facts. I want the memory to get closer to reality so I can watch how things evolve over time. I want a knowledge graph.

Everyone agrees knowledge graphs and GraphRAG provide a more performant substrate for a unified agent memory layer than plain RAG. But kicking one off is far harder. The resistance always collapses to the same wall: how you model your data. Your ontology is the hardest part of the system.

If you can’t define your ontology properly for your domain, the graph won’t represent the reality you want. The right entities and relationships simply aren’t there. As a result, GraphRAG ends up performing worse than the simple RAG you were trying to beat.

This translates straight to a memory layer. There’s no dodging it. Even if you stay file-only (a “virtual knowledge graph,” like an LLM knowledge base over your notes), you still hit the same data-modelling question: which primitives, and which entities, do you even extract?

The instinctive reaction is to design the perfect, complete ontology upfront. That’s exactly the trap that freezes the project.

The strategy is a not-overkill ontology. You need something flexible enough to kick off with almost no friction before you really know your domain, extending it with domain-specific detail as you explore your data.

Concretely, you use a small, fixed, generic, but extendable noun data model, known as POLE+O. Plus two core primitives, Preferences and Facts, for everything that doesn’t fit into the nouns.

You ship something that works, then add subtypes as a lightweight data-exploration step shows you where the generic types clash with your real data.

This approach lets you stand up a knowledge-graph memory layer for your own assistant without burning weeks on schema design. To build this, we first need to understand what an ontology actually is and why targeted models beat exhaustive ones.

Start Your Transition Into AI Engineering (Product)

This article showed how to design the ontology your knowledge-graph memory needs. My Agentic AI Engineering course shows the harness around it. I just released a free preview to build and run a working agent in 5 minutes.

You build a multi-agent system with two MCP servers (Research Agent + Writing Workflow), a deep research algorithm, an evaluator-optimizer loop, observability, and LLM-as-judge evals. Patterns required to ship AI.

Built for software, data engineers or scientists transitioning into AI engineering.

7 free lessons, 2 MCP agents ready for your GitHub portfolio. Part of our 35-lesson course. Rated 5/5 by 300+ students.

Start the free preview →

What Is an Ontology?

An ontology is the formal answer to 1 question. When you read the world, what do you write down as nodes, and what do you draw as edges? It specifies the kinds of things that exist in your domain, their properties, and how they relate to each other.

The ontology’s job is to map a targeted slice of the real world into the digital world. A good ontology is highly targeted to the problem you actually want to solve. If you over-model, you drown in noise and never ship. Plus, it get’s extremely expensive to extract and maintain the knoweldge graph. If you under-target, the graph doesn’t reflect the reality you care about.

Look at concrete, shipped ontologies for real-world proof. The create-context-graph domain catalog made by Neo4j publishes 22 ready-made domain ontologies. Every single one lands at exactly 10 to 12 entity types. They use a shared 5-noun base plus only 5 to 7 domain-specific nouns.

For example, the Personal Knowledge domain models the world as Note, Contact, Project, Topic, Bookmark, and JournalEntry. The Agent Memory uses Agent, Conversation, Memory, ToolCall, and Session. The lesson here is that real ontologies are small on purpose. They capture only the entities required to answer the questions the system is designed for.

So if targeted and small is the goal, why does everyone — me included — reach for big and perfect first? That’s the trap.

The Overkill Trap: Why My Knowledge Graphs Never Shipped

When I first encountered the ontology concept, I assumed I had to study my domain in depth. I thought I needed to model all of finance, for example, and design the ideal ontology before working with any real data. You can’t actually do that before you have a system running and data to look at. You just pile up assumptions that mostly turn out wrong.

I got frozen. Every knowledge-graph solution I started stayed on my laptop and never got used, because I was waiting on an ideal ontology I could never reach. Without understanding the ontology, I couldn’t even write a decent extraction step to populate it. I was deadlocked, bringing 0 value.

The breakthrough was realizing I need a couple of models that let me start generic and extend over time. As I get more data, analyze it, and actually understand my problem, the schema evolves. Let’s meet the base model that lets you start in 5 minutes instead of 5 weeks.

The POLE+O Data Model

POLE+O is a tiny, fixed, top-level vocabulary that can classify almost anything you pull out of text. It stands for Person, Object, Location, Event, and Organization [2]. It originated in law-enforcement and intelligence analysis. The Organization type was added for general-purpose entity extraction. The point of a fixed base is queryability. There are always exactly 5 base nouns to filter on, so the graph stays answerable no matter how it grows underneath.

Person covers people, aliases, and personas. Object covers physical or digital things. Location covers places, addresses, and regions. Event covers meetings, transactions, and incidents. Organization covers companies, teams, and institutions. Two or three of these catch the overwhelming majority of what a personal assistant needs.

Here are POLE+O’s five base types and the default subtypes each one ships with:

Here’s the beauty of this approach. You extend the base nouns with your own subtypes, and that’s how you tailor a generic ontology to your specific domain. It works exactly like object-oriented programming. You start from base classes you adopt without thinking. Then you subclass into specifics as your use case clarifies.

You can kick off with nothing extended and add concrete types only as you understand your data better. Neo4j’s agent-memory library uses precisely this approach. POLE+O is its default, swappable ontology.

The data-exploration workflow runs in a simple loop. First, kick off with generic POLE+O. Second, run an exploration extraction over your real data. Forget production reliability. You only care about understanding what’s there. Third, inspect the graph for clashes where the generic model lies about your data. Fourth, add or rename subtypes to fix each clash. Finally, repeat the process. You won’t get it perfect, and that’s the point. You iterate like any other AI app instead of freezing.

Look at named examples from real extraction runs. Claude Code comes back tagged as a Person when it’s clearly an Object. The “AI Engineer” conference lands as an Event when you wanted an Organization. DeepSeek is tagged a Person, not an Object.

Portugal and New York both get a flat Location label even though one’s a country and one’s a city. An agentic harness shows up as a generic Object when, for knowledge work, you’d rather have a Topic type. Each clash is a signal to add 1 subtype, not to redesign the whole schema.

POLE+O nouns and their subtypes cover the things in your world. But to fill in the gaps there are two specials tricks we have to go over.

Preferences: The Things a Noun Likes

Preferences are the second family of entities you attach to the graph. They are things a noun likes or dislikes. A Preference is a characteristic of an entity. It represents a stance. The canonical case is a person who likes, prefers, or dislikes something.

Concretely, a Preference entity looks like this:

category groups the preference, preference is the statement itself, and context optionally records when or where it applies. confidence runs from 0 to 1. The embedding makes it semantically searchable.

Make it concrete. “Loves Italian food”, “prefers dark mode”, and “dislikes long meetings” are clear examples. Each is a stable stance the assistant should remember and adapt to.

By default, a Preference hangs off the Person. That’s the most common and useful case. You can extend preferences to other objects, like an Organization’s policies, a car’s settings, or an Event’s dress code.

Because I’m building a personal assistant, I start by attaching Preferences only to the Person. This keeps the graph clean, low-noise, and small. I’ll extend it later only when a concrete use case demands it.

Preferences are the personalization layer. They act as the memory of the user’s stances. They are the “sweet sauce” that makes every future response feel tailored.

There is one issue. Plenty of useful knowledge is just an atomic fact. Forcing all of that into the ontology is how graphs explode in complexity. The fix is a deliberately generic primitive.

Facts: The Trick You Haven’t Thought Of

The Facts entity is the fallback for everything that doesn’t cleanly fit a noun or a Preference. You drop the claim into a generic Fact. This is the move that keeps the ontology small and stops you from over-thinking the schema.

A Fact is the closest thing to a classic-RAG chunk. An LLM produces each Fact during extraction. Each Fact holds a single, atomic concept which works like a charm via semantic search.

The beauty is that with facts you avoid the usual chunking errors, such as splits mid-thought, mixed concepts, and arbitrary boundaries. In reality, a Fact is a triplet. A subject, predicate, and object like “Eiffel Tower / is / 330m tall” gets embedded and stored as 1 granular unit.

Here is the shape of a Fact entity:

The triplet — subject, predicate, object — is the whole fact. valid_from and valid_until give it optional bi-temporal validity. The embedding, computed over the concatenated triplet, is what makes the fact retrievable by semantic search.

It’s confusing that we have a triplet stored as a node. But this is what it makes it flexible. We don’t worry about modeling these one-off triplets directly into the ontology, but the LLM extracts them as-is from the text.

Facts are usually wired to nothing. They have no relationships to other entities. They are retrieved only via semantic search and text search. A Fact stays in the graph but is independent of it. This works because a graph store runs vector search and graph traversal in the same query engine [4]. Which means facts are retrieved only via semantic/text search.

Facts let you ship a memory layer before you have the perfect ontology. Anything you can’t yet model degrades gracefully into a searchable atomic node instead of blocking the build. Early on, you lean on Facts. As the graph matures, claims migrate toward typed entities and edges. It costs nothing to schema and nothing to maintain when entities merge or get deleted.

What’s Next

The takeaway is the posture. An ontology is a living artifact you bootstrap from a fixed generic base and grow through a data-exploration loop, exactly like any other AI application.

If you want to see the whole strategy implemented, the fastest path is to play with Neo4j’s agent-memory SDK or its MCP server. It uses POLE+O as a swappable default, subtypes as cheap extensions, and Preferences and Facts as first-class primitives. Studying it is what made all of this finally click for me.

I’m actively migrating my own Obsidian Second Brain toward the POLE+O, Preferences, and Facts primitives. This turns thousands of files into a graph I can actually traverse, visualize, and watch evolve over time.

But here is what I’m wondering:

If you worked with Knowledge Graphs, what was your process in discovering your own ontology?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

References

1. Create Context Graph. (n.d.). Domain Catalog. create-context-graph. https://create-context-graph.dev/docs/reference/domain-catalog

2. Neo4j Labs. (n.d.). POLE+O Data Model. Neo4j Agent Memory. https://neo4j.com/labs/agent-memory/explanation/poleo-model/

3. Neo4j Labs. (n.d.). Neo4j Agent Memory. GitHub. https://github.com/neo4j-labs/agent-memory

4. Neo4j Labs. (n.d.). Why Neo4j? Graph-Native Memory Architecture. Neo4j Agent Memory. https://neo4j.com/labs/agent-memory/explanation/graph-architecture/

Images

If not otherwise stated, all images are created by the author.

I already have a second brain setup based on Obsidian, Readwise, NotebookLM, and Claude Code. I dump all my notes, research, and highlights there. Whenever I want to create content, I create a scoped wiki targeted toward the topic. I gather information from my second brain using a deep research algorithm on top of my private data and external resources via NotebookLM. The wiki is structured like the LLM Knowledge Base presented by Andrej Karpathy.

This setup fails to extract and maintain shared entities, preferences, and facts across the wiki as the knowledge base grows. For example, if the topic “Claude Code” is mentioned in 10 documents, I want to extract all the metadata about it into its dedicated folder. I want to see what other entities it relates to, such as Anthropic, San Francisco, Codex, or Gemini CLI. I also want to see how many documents mention it to rank frequency. You can do that with a pure file-based system and Obsidian, but performance degrades when your data scales past 50 documents.

The same concept applies to any unstructured knowledge base. You need a way to extract and connect knowledge from your conversations, documents, and images. This becomes essential between conversations so your agent doesn’t forget you. Instead, it provides a personalized experience. It’s also critical for context engineering to inject the right context at the right time and keep the LLM focused on relevant facts.

Most teams default to one of two memory approaches. Both collapse under real use. A file system gives you append-only logs that the agent re-reads from scratch, which fragments and rots context.

A vector index gives you fuzzy semantic recall but no merge, no identity, and no way to know if this is the same Karpathy you knew yesterday. Durable AI memory requires a structured graph to track identity and relationships [1]. Without this structure, the assistant forgets past interactions and fails to build compounding intelligence.

Knowledge-graph memory is the next step on the arc from Retrieval-Augmented Generation (RAG) to agentic RAG to agent memory [2]. Building a unified knowledge-graph memory system is hard, so most teams skip it.

During my research, I stumbled upon neo4j-labs/agent-memory. It’s a masterpiece. Who knows more about knowledge graphs (KGs) than Neo4j?

After I spent 2 days playing with it and understanding the codebase, I realized it was the perfect mental model for any agent memory system powered by KGs.

In this article, I’ll walk through the core architectural patterns of neo4j-labs/agent-memory. It features 1 graph, 3 memory tiers, the POLE+O ontology, a 3-stage extraction pipeline, a composite resolver, and the SAME_AS pattern.

By the end, you’ll have a concrete mental model. You can ship on top of their Software Development Kit (SDK) or hook it into your agent via their Model Context Protocol (MCP) server. Alternatively, you can steal the patterns and ship the same architecture on Postgres or MongoDB if a full graph database in production doesn’t make sense for your use case.

Start Your Transition Into AI Engineering (Product)

This article shows the memory layer your agent needs. My Agentic AI Engineering course shows the harness around it, and I just released a free preview that lets you build and run a working agent in 5 minutes.

You build a multi-agent system with two MCP servers (Research Agent + Writing Workflow), a deep research algorithm, an evaluator-optimizer loop, observability, and LLM-as-judge evals. The production patterns behind agents that actually ship.

Built for software, data engineers or scientists transitioning into AI engineering.

7 free lessons, 2 MCP agents ready for your GitHub portfolio. Part of the 35-lesson course. Rated 5/5 by 300+ students.

Start the free preview →

What’s Inside neo4j-labs/agent-memory

The SDK takes natural-language interactions on the write side and returns a fused memory context on the read side. Everything anchors to a single Neo4j graph. For our scoped wiki, notes and Readwise highlights about Claude Code flow in. A structured pull of what the agent knows about Claude Code, how it relates to Anthropic, and its frequency across 50 documents comes out.

At its core, there is 1 graph and 3 memory tiers joined by typed edges: short-term conversations, long-term typed entities, and reasoning traces. They’re stitched together by :MENTIONS, :TOUCHED, and :INITIATED_BY relationships [3].

The architecture contains 8 small, single-responsibility modules. The models/ module holds Pydantic schemas. The schema/ module handles Cypher migrations. The extraction/ module runs the Named Entity Recognition (NER) pipeline. The resolution/ module holds the composite resolver. The dedup/ module manages the SAME_AS pattern. The core/ module provides MemoryClient.get_context(). The mcp/ module runs the FastMCP server with 15 tools. The integrations/ module holds 9 framework adapters for tools like LangChain and LlamaIndex.

Consider an end-to-end scenario. You drop a Readwise highlight about Claude Code into your scoped wiki. The extraction/ module pulls Claude Code as an Object, Anthropic as an Organization, and Codex as an Object. The resolution/ module canonicalizes each against existing nodes. The dedup/ module checks vector similarity and either auto-merges or flags a pending :SAME_AS edge. The schema/ module commits :MENTIONS edges from the note to each entity.

Later, MemoryClient.get_context() pulls fused context across the same graph in one call. This matters concretely for the scoped-wiki agent. You can ask what you discussed last session, what you know about Claude Code, and why the agent surfaced a Codex comparison last Tuesday. The SDK answers all three against the same graph. It uses the same Cypher dialect with no cross-store join logic.

Short-Term, Long-Term, Reasoning Memory

The SDK splits memory into three layers that all live on the same Neo4j graph [3]. Short-term memory is the linear message sequence. It uses ordered :Message nodes chained by :NEXT edges, scoped to a :Conversation. Long-term memory is the typed entity graph. It uses deduplicated :Entity nodes with vector embeddings and arbitrary domain relationships.

Reasoning memory is a tree per agent run. It uses a :ReasoningTrace root with child :ReasoningStep nodes capturing thoughts and tool calls. For the scoped-wiki agent, short-term memory holds your current chat. Long-term memory holds the canonical Claude Code entity plus its relations to Anthropic, San Francisco, Codex, and Gemini CLI. Reasoning memory holds the trace of how the agent picked those specific notes to answer you.

Three relationships do the entire stitching. The :MENTIONS edge joins short-term to long-term memory. The :INITIATED_BY edge joins reasoning to short-term memory. The :TOUCHED edge joins reasoning to long-term memory. These three edges make provenance a one-hop query rather than a log-reconstruction project.

Reasoning memory is the novelty from this architecture. By storing past successful or failed thinking patterns into the memory, the agent can one-shot future similar requests or at least know not to repeat similar mistakes. Intuitively, it’s similar to Reinforcement Learning (RL), but instead of baking the optimizations into the weights, you do it at the database level.

The most important part of this architecture is the ontology.

The Ontology

The long-term memory uses a closed five-type vocabulary for its ontology known as POLE+O. It uses Person, Object, Location, Event, and Organization, borrowed from intelligence-analysis taxonomies [5]. Every entity is exactly one of these five types. Subtypes are open, but the top-level vocabulary is fixed.

In the personal assistant, Karpathy is a Person. Claude Code is an Object. Anthropic is an Organization. Your Tuesday deep-research run is an Event. San Francisco is a Location.

Type and subtype materialize as multi-tier Neo4j labels. The query builder sanitizes and PascalCases them into labels like :Entity:Person:Individual. You can search by type or subtype, making this solution highly efficient.

Using this strategy, you can extend each core type from POLE+O with your own custom domain. Other defaults are: :Entity:Location:City, :Entity:Event:Concert, :Entity:Organization:Company, etc. Here is a catalog of over 20 domains such as Data Journalism, Gaming, Personal Knowledge, and Product Management.

Entities modeled via POLE+O are nouns. The SDK adds 2 other node types beyond entities.

:Fact nodes hold every claim mentioned in the text. They’re intentionally generic so the ontology doesn’t get over-specified. They serve as a fallback when nothing else fits. You can intuitively see them as chunks of text that contain only 1 concept.

Then there are :Preference nodes that store user preferences via a SUPERSEDED_BY relationship. As agent memory is user-centric, this provides the WOW effect where the agent remembers past preferences and learns from them over time.

For the scoped wiki, “Anthropic developed Claude Code” is an edge. “Claude Code 1.0 shipped in 2025” is a :Fact. “I prefer agent-harness comparisons over pure benchmarks” is a :Preference.

Extraction: From Raw Text to Typed Entities

The SDK runs entity extraction as a speed-versus-accuracy ladder. It uses spaCy for fast statistical NER. It uses GLiNER and GLiREL for zero-shot extraction. It uses an LLM stage for cases that need real semantics and to extract the relationships between them [6].

Each stage maps its outputs back to POLE+O types. It uses explicit merge strategies when 2 extractors disagree. When you drop a Readwise highlight about Claude Code into your scoped wiki, spaCy lifts proper nouns like Anthropic and San Francisco. GLiNER catches domain entities like Claude Code and Gemini CLI. The LLM stage only fires when the previous 2 stages leave ambiguity, or when the model needs to extract relationships.

Routing every mention through an LLM would multiply extraction cost massively for marginal recall on rare entities. The ladder pushes high-confidence cases to cheap models. It escalates only ambiguous mentions to the zero-shot models and reserves the LLM stage for when real semantics matter.

The real problem is at the normalization step.

When Two Mentions Are the Same Entity (And When They Aren’t)

Resolution and deduplication are 2 different problems. Resolution sets a canonical string property on an existing reference. Deduplication decides whether a new node gets created at all. Conflating them is how graphs end up with 3 Anthropic nodes that none of your queries find together [7].

Resolution runs 3 strategies on the name field in cost order. Exact matches existing canonical strings. Fuzzy uses RapidFuzz string similarity for surface variants like “A. Karpathy” and “Karpathy, Andrej”. Semantic falls back to embedding similarity for cases like “the founder of Eureka Labs”. It only matches between nodes of the same type, meaning a Person only resolves against Person candidates.

After resolution runs, two mentions like “Apple” and “Apple Inc.” end up with different surface names but the same canonical name. That’s why a second step is needed. Deduplication looks at the semantics, not just the name.

For deduplication, the SDK uses vector and fuzzy similarity across the entire node content. This ensures the node is actually the same, not just a name coincidence. In other words, this avoids false positives. Using vector and fuzzy search, the SDK computes a score.

Scores at or above 0.95 trigger an auto-merge. Scores below 0.85 create a new node. Scores between 0.85 and 0.95 don’t silently merge. Instead, they create a :SAME_AS edge with a pending status. This flags the edge for a human or downstream agent to resolve later. This pattern stops “Jensen Huang the NVIDIA CEO” from merging with “Jensen Huang the Taipei dermatologist” just because their embeddings landed 0.91 apart [7].

A false merge is silent and unrecoverable. A false split is noisy but recoverable. You can’t undo a false merge without re-ingesting from the raw source data. That’s why you should leave uncertainty to a human.

Zooming into the Retrieval Algorithm

Because all three tiers live on one graph, a single retrieval can compose vector similarity over :Entity embeddings, multi-hop expansion over typed relationships, time-ordered :NEXT conversation walks, and reasoning-trace lookups via :INITIATED_BY and :TOUCHED joins. All of these run as steps in the same Cypher query. Neo4j 5.20 introduces db.index.vector.queryNodes, making vector similarity a first-class graph operation [4].

When you ask what you know about Claude Code, how it relates to Codex and Gemini CLI, and why you looked at it last week, the agent fuses three things in one pull. It uses vector similarity over your Readwise highlights to surface relevant passages. It uses a multi-hop traversal of :DEVELOPED_BY and :COMPETES_WITH edges to bring in Anthropic and Codex neighbors. Finally, it uses an :INITIATED_BY jump back to the prior conversation that discussed agent harnesses. There’s no cross-store join logic and no orchestrator.

From our tests, the library leaves the context construction to the user of the SDK. In other words, you get the whole output from the graph, and it’s your responsibility to further compress it before passing it to the LLM.

What’s Next

The neo4j-labs/agent-memory architecture is more complex than what this article covers, but this is the core idea behind it. I’ll cover other components in more depth in future articles, including designing the ontology and keeping your knowledge graph clean over time.

I think this open-source repository is a perfect blueprint you can take to build your own agent memory solution, even with Postgres or MongoDB, to avoid keeping multiple databases in production. Still, Neo4j is probably the best choice for data mining and exploration.

For small to medium-scale projects with thousands of nodes and short hop traversals, I’d probably build my own agent memory solution from scratch on top of Postgres or MongoDB. I’d reach for Neo4j as an internal tool within my organization, or when the scale or complexity becomes too large for Postgres or MongoDB.

But here is what I’m wondering:

How are you handling agent memory today? Flat files, a vector index, a knowledge graph, or something stranger?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

References

1. Seale, T. (n.d.). This week Anthropic dropped Claude Sonnet 4.5. LinkedIn. https://www.linkedin.com/posts/tonyseale_this-week-anthropic-dropped-claude-sonnet-activity-7379787334398926848-iVOE/

2. Monigatti, L. (n.d.). The Evolution From RAG to Agentic RAG to Agent Memory. Leonie Monigatti. https://www.leoniemonigatti.com/blog/from-rag-to-agent-memory.html

3. Neo4j Labs. (n.d.). Understanding the Three Memory Types. Neo4j Agent Memory. https://neo4j.com/labs/agent-memory/explanation/memory-types/

4. Neo4j Labs. (n.d.). Why Neo4j? Graph-Native Memory Architecture. Neo4j Agent Memory. https://neo4j.com/labs/agent-memory/explanation/graph-architecture/

5. Neo4j Labs. (n.d.). POLE+O Data Model. Neo4j Agent Memory. https://neo4j.com/labs/agent-memory/explanation/poleo-model/

6. Neo4j Labs. (n.d.). How Entity Extraction Works. Neo4j Agent Memory. https://neo4j.com/labs/agent-memory/explanation/extraction-pipeline/

7. Neo4j Labs. (n.d.). Entity Resolution and Deduplication. Neo4j Agent Memory. https://neo4j.com/labs/agent-memory/explanation/resolution-deduplication/

Images

If not otherwise stated, all images are created by the author.

I needed a TypeScript harness for my latest book code. It required a Terminal User Interface (TUI), an agent loop, tools, Model Context Protocol (MCP) support, skills, and slash commands. I will be honest with you. I first tried to vibe code this project.

As I knew what I was looking for, it worked. Until it didn’t. The code was working until you started looking more closely at the details. Only the first 20 characters were rendering inside the TUI, and the skills weren’t invoked by the agent loop.

So I deleted the whole code base and started over with a new strategy.

The cost of vibe coding isn’t abstract. It’s the next feature you can’t ship because you’re debugging a slash-command renderer that looked finished. This is what most people get wrong. Output that compiles and looks done breaks the moment you reach for the rough edges.

I divided the harness into tasks. I one-shotted the barebones version, which was just a TUI plus an agent loop with bash, grep, and a todo tool. Then I layered MCP, skills, and slash commands as separate features.

You can’t one-shot whole applications. You can one-shot big features if you scope them right and run them through a real engineering process.

This is known as agentic coding. Not vibe coding. You’re using agents to write the whole codebase, but you are still the mastermind behind everything.

But I wanted more. I wanted to automate this process. But with a single constraint in mind: “the code should HAS to be good”.

That’s why I built Squid. It’s an opinionated six-agent Claude Code setup available at iusztinpaul/squid. It ships features the way a real software team ships them.

Squid has already shipped our content-automation tool, expanding it from articles to posts, notes, threads, and messages. It shipped the book’s code data pipelines and TypeScript harness.

In this article I will show you how it works.

The concrete blueprint relies on a specialized team and an e2e lifecycle.

Start Your Transition Into AI Engineering (Product)

Squid applies the multi-agent pattern to coding. My Agentic AI Engineering course applies it to writing, and I just released a free hands-on lesson that distills the whole system.

You build a multi-agent system composed of two FastMCP servers (Deep Research + LinkedIn Writer) orchestrated by a harness, plus an observability and evals layer on top. The shift from classic backend/frontend stacks to MCP servers and harnesses is the pattern shaping modern agentic AI.

Built for software and data engineers moving into agentic AI engineering.

Part of the 35-lesson course. Rated 5/5 by 300+ students. First 7 lessons free.

Start the free lesson →

The Six Agents Engineering Team

The system contains six agents. No agent both writes code and decides whether the code is correct.

The product manager agent manages the tasks and ensures the feature adheres to the software architect’s specifications. It takes a raw feature specification, writes or updates an Architecture Decision Record (ADR) for non-obvious choices, and splits the feature into ordered tasks. It also maintains the Domain-Driven Design (DDD) glossary so vocabulary stays consistent between the business and engineering.

Note how, because Claude can easily handle both PM and software architecture work, we decided to merge these roles together. We did this to avoid fragmenting the context just to follow a standard human process. Ultimately, planning should be closely aligned with the software architect’s vision. In human processes, dividing these two responsibilities often created more issues than solutions.

The software engineer agent uses red-green Test-Driven Development (TDD). It writes the failing test, writes the minimal code to pass it, and then refactors. The software engineer uses direct command-line interfaces (CLIs) like git, mongosh, and gh. It never uses MCP wrappers. CLIs are more flexible because they tap directly into the power of bash. Plus, LLMs have seen considerably more bash code than MCP wrappers during training.

The tester agent specializes in the adversarial end-to-end edge-case pass. It catches false-confidence claims where the software engineer says the tests pass. It does this by reading every acceptance criterion against concrete evidence, like the test name, file lines, and command output.

The pull request reviewer agent performs a diff-only review. It looks for dead code, duplication, missing test coverage, and documentation adherence. It does a narrow performance review on hot paths only. It’s explicitly told not to micro-optimize one-off scripts.

The on-call agent loops on the Continuous Integration (CI) pipeline until it passes. In an earlier iteration, the CI check lived inside the software engineer and tester loop, and it got skipped constantly. Promoting it to a dedicated agent invoked by the orchestrator increased the probability the step runs.

The self-improve agent is an optional meta agent. After the feature is done, while looking over the results, the human can run the self-improve agent to scan the run for high-signal lessons and propose updates to the agentic coding layer that consists of CLAUDE.md, skills and subagents. This is a double-edged sword. It can constantly improve your workflow or quickly degrade it if you are not careful. That’s why it’s incredibly important that this step is gated by a human.

The secret sauce is in anchoring the agents into your own documentation.

Keeping Up With Documentation: ADRs & DDD Glossary

The ADR directory acts as compressed architectural memory across runs. Every non-obvious choice regarding the datastore, synchronization defaults, authentication boundaries, or dependency lock-in ships with an ADR. These records include the status, context, decision, and consequences at docs/adr/<NNNN_title>.md. The product manager reads the directory before grooming a new feature, so decisions stay consistent across feature branches.

The DDD glossary gives shared vocabulary between the business and engineering at docs/glossary.md. It enforces one canonical name per concept. Code identifiers, OpenAPI schemas, database columns, and customer-facing interfaces all use the term exactly as it appears there. This gives Claude Code business context, not just code context, properly anchoring your code in your domain. The software engineer, tester, and pull request reviewer all reason about the same domain.

I have an honest caveat. The agents still under-use both the ADRs and the glossary. The spine exists, but I am still working on getting the agents to lean on it consistently.

Now the agents have the context they need to execute a feature from a raw specification all the way to a merged pull request.

The Night Skill. The End-To-End Workflow.

The /night skill takes one input, which is a feature specification written by the human, and produces one output, which is a merged pull request with green CI. Everything in this section sits between those two endpoints.

The /night pipeline is a long-running lifecycle. That’s why it’s called the “night” skill. It’s scoped to run for hours at a time, often with multiple pipelines in parallel.

It has two human checkpoints and five retry caps, while everything else is automated. The orchestrator acts as a manager. It never writes code itself, never runs tests itself, and never reviews the diff itself. It launches agents and enforces human validation.

After a human carefully writes a detailed feature specification, it calls the /night skill, which creates a new branch and worktree. The product manager reads the glossary and ADR directory, updates or writes a new ADR if needed, and splits the feature into a task plan.

Then we hit the first human gate. The user approves the plan, optionally sharpened by the /grill-me skill. The /grill-me skill is inspired by Matt Pocock’s work, which forces the agent to ask sharp questions back about anything fuzzy in the plan, such as interfaces, modularization, or new tools. This conversation is the line between vibe coding and agentic coding.

Next is the inner loop per task. The software engineer implements the code, the tester verifies it, and failures route back to the software engineer. This loop is capped at 5 attempts. Convergence is mostly mechanical through a run, fail, fix, and run cycle.

The product manager then performs an acceptance review on the whole feature from the user’s perspective. Rejections are packed into a single task back into the inner loop. This is capped at 3 attempts, because judgment-call loops are where Claude Code spirals.

Next, we repeat a similar loop using the PR reviewer agent, which looks at the diff, with a maximum of 3 attempts to avoid perfectionism. Adding a maximum number of attempts here is critical, because during review an LLM almost always has something else to say.

After the push, the on-call agent watches CI with a maximum of 5 attempts, routing failures back to the software engineer.

When the CI is green, we notify the user (e.g., via Slack) that the PR is ready for review. Optionally, based on any potential issues found while running the /night skill, we run self-improve to propagate that into your memory.

The /night lifecycle. Two human gates, five retry caps, everything else automated.

Beautiful! With this process I one-shot most of the features I am working on. And when it’s not a one-shot, I’m typically 95–99% there by the time I review the PR.

How the Tester Stopped Re-Running What the SWE Already Ran

The biggest problem with the e2e workflow above is that it’s slow and redundant. I preferred that over generating AI slop that I have to manually review and fix.

Still, there are a few tweaks that we can make to the workflow to improve speed and efficiency.

For example, when the tester re-ran the linter, type checker, formatter, and the happy-path suite that the software engineer had already run, we paid for everything twice. This was the number-one source of having a system that works but is too slow to use.

To fix this, the tester now accepts the software engineer’s reports for formatting and happy-path tests. It only runs the adversarial end-to-end edge-case pass itself. This covers the part the software engineer can’t credibly self-verify. Trust is bounded. Intuitively, I realized I’d started shifting the Tester toward QA-style practices, rather than just running simple tests.

I am still iterating on optimizations. For example, I want to route some subagents to Claude Sonnet models instead of Claude Opus. I also plan to narrow toolsets per role to reduce reasoning failures.

Also, depending on what you are working on, you might want to use the system more as a fast, snappy assistant than as a long-running workflow that prioritizes correctness above all.

Day vs. Night: Two Orchestrators, One Team

That’s why we have two pipelines running the same agents. The /night skill is the full lifecycle. It’s long-running, set-and-forget, has two human gates, and runs while you are away from the keyboard or working in parallel.

The /day skill is the lean inner loop. It runs the software engineer, the tester, and human commits for surgical edits. It skips product manager grooming, the pull request reviewer, and the on-call agent.

There is a concrete use case for the /day skill. When I read a merged pull request and find code I don’t like, the /day skill runs the stripped software engineer and tester loop to apply targeted edits. Then the on-call agent cleans up any CI fallout. This is the surgery that keeps the system from becoming a black box.

Both pipelines have one thing in common. The human is in the loop on purpose, not as a fallback.

Why Code Templates Are a Waste of Time in 2026

Most teams are still scaffolding from cookiecutter templates that were outdated the day they were committed. This is a maintenance tax disguised as productivity. Squid stops paying that tax. Technology moves fast enough that any frozen template’s frameworks, tooling, interfaces, and opinions all need their own maintenance pipeline. That’s only worth it if one template fans out across dozens of projects.

A Copier or cookiecutter template isn’t free. I tried scaling one across Python, TypeScript, and Go. I watched the project balloon into a maintenance burden where most files would never be used. Maintaining a template engine to support multiple stacks is a full-time job.

Asking Claude Code to copy from the last project fails too. It propagates the technical debt baked into the source codebase. You inherit the mess, not the ideal state.

The real shift relies on markdown, not Jinja. I call these agentic templates.

You encode good practices as skills and CLAUDE.md files. Fundamentals like clean architecture, CI/CD discipline, testing patterns, and development cycles rarely change. When they do change, you edit prose instead of regenerating from a template engine that quickly slides into dependency hell.

Tooling stays dynamic. You don’t pin framework versions inside a template. You keep a decision tree of allowed choices and let the agent pull the latest interfaces on demand via Context7 at scaffold time.

Project structure can’t be templatized. The anti-pattern organizes by type, putting files into agents/, nodes/, schemas/, and tools/ directories. One business module’s logic ends up scattered across four folders, forcing both humans and the agent’s context window to thrash.

The correct pattern organizes by actionability, keeping one bounded context per directory. Each domain owns its own types, store, Application Programming Interface (API), and prompts. That’s locally readable, easier to maintain, and easier for the agent to reason about.

Because we describe the structure in Markdown files instead of cookiecutter templates, we can define it like this:

Avoid global dumping grounds like utils/ or helpers/. Avoid a root-level types.py grab bag. Avoid grouping tests by type.

The /scaffold skill acts as an interactive bootstrap. An AskUserQuestion prompt drives a tight decision tree covering project identity, layout, components, backend, frontend framework, infrastructure, agent team, tracker, ADR and glossary opt-ins, and external services. A deterministic table picks only the matching specifications from the specification library. Unused categories never enter the context. The skill writes a tailored CLAUDE.md brief, lays down an empty folder skeleton, and hands off.

Then, based on the agentically generated template, you can use /night or /day to start writing real code.

Open-Sourcing Squid

I don’t want to keep Squid for myself. I want to share it with the community to learn from and contribute to.

Thus, I am open-sourcing Squid.

You can install it as a Claude Code plugin:

/plugin marketplace add iusztinpaul/squid
/plugin install squid@squid

I want you to try it, build something awesome with it, and if you like it, contribute back:

Check the full codebase

Still, here is what I’m wondering:

What is your agentic coding setup? How is Squid different from your own approach?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

Images

If not otherwise stated, all images are created by the author.

I gave this talk twice in one month: at O’Reilly’s Context Engineering Event and at Abi Aryan’s Maven course on LLM inference at scale. After being blasted with questions, I realized something: GraphRAG isn’t a retrieval algorithm, it’s a data modeling problem.

Powering agents with knowledge graphs (KGs) and ontologies is still an unsolved problem. All the engineers I spoke to want GraphRAG, but don’t know how to implement it.

But at its core, we should ask a different question. Why do we even need GraphRAG in the first place? Why complicate our solution over a simple RAG system?

There are three core reasons.

First, you face context rot. As the context window fills, the signal-to-noise ratio collapses. The LLM degrades.

You pay for this degradation in quality, cost, and latency [1].

Second, you face data fragmentation. In the agent era, your data lives in silos most builders share: documents, notes, research, emails, and text messages. We are no longer lucky enough to have all the data nicely stored in a single database.

Third, the agent’s unified memory naturally maps to a knowledge graph (KG). People have preferences and experiences. They went into specific locations, met with other people, or have a list of items to do. Things get trickier when “Arthur told Felix that his favorite coffee shop is in the center of Timisoara”, but after two months “it moved to Lisbon”. You need to start tracking relationships between people, locations, and most especially how these relate in time.

GraphRAG solves all three.

This is a data modeling problem, not a retrieval algorithm. It took a painful LangChain detour and a hard MongoDB RAM conversation to settle that for me. You need an ontology.

By the end of this article, you will learn about ontology-first design, the three extraction modes, append-only data models, and hybrid retrieval joined by Reciprocal Rank Fusion (RRF). Finally, you will see how to expose the GraphRAG engine as a unified memory layer via an MCP server to power your agents. In other words, how to do agentic GraphRAG.

Before walking through the architecture, let’s understand why the story has to start from the ontology.

Build Your Own Multi-Agent System Free Workshop (Product)

This article shows what an MCP-served unified memory looks like end to end. If you want to actually build agentic systems with MCP servers like this, I open-sourced a hands-on workshop for that.

Two MCP servers from scratch: a Deep Research Agent (Gemini + Google Search grounding) and a Writing Workflow with an evaluator-optimizer loop.

Packaged with slides, a ~2-hour video, runnable reference code, and an “implement-it-yourself” skeleton via agentic coding best practices (25 tickets, one orchestrator skill, and two agents: SWE and tester).

Originally presented at the AI Engineering Conference Europe. 200+ stars on GitHub. Free.

Go to workshop →

Why the Story Starts From the Ontology

Whenever you need to connect dots across a corpus of multiple documents rather than find the most relevant paragraph, you go for GraphRAG. Knowledge is stored as entities and edges.

You traverse connections rather than find similar text.

An ontology is a collection of classes and the relationships allowed between them. If you come from object-oriented programming, you already have the right intuition.

Throughout this article, we will build a digital twin. My favorite example. We will define a Global Ontology of six entity types organized into two sub-ontologies.

The data pipeline deterministically constructs the Document Ontology. It contains DOCUMENT and CHUNK nodes. It uses PART_OF, NEXT, REFERENCED, and MENTIONS edges.

The LLM extracts the Person Ontology. It contains PERSON, TASK, EPISODE, and PREFERENCE nodes. It uses RELATED_TO, TODO, EXPERIENCED, and HAS edges.

The schema is flexible. You define it for your business case. Every section after this one assumes these exact node and edge labels.

Skipping the ontology carries a heavy cost. I tried LangChain’s MongoDBGraphStore, which lets the LLM extract entity and relationship types freely. Five documents produced 17 node types and 34 relationship types.

This included part_of, Part Of, and part of as three separate types. The underlying data model does not enforce a schema at the storage layer.

With an ontology, the LLM can only extract what you defined. The constrained scope also allows you to use cheaper extractor models.

That’s why GraphRAG is the right tool when you have a clearly defined schema. It works when you need to identify relationships.

It reduces hallucination on complex queries that span interconnected facts. Domains where knowledge graphs naturally fit are legal, medical, financial, business operations, productivity tools and in my opinion, the crown jewel: personal assistants. With a KG, you can naturally build the unified memory of your personal assistant to properly remember what you like, what you did, and what you have to do, all anchored in time.

For example, Palantir built its empire using ontologies. Google uses KG to power its search, and Microsoft uses it in its internal ops tools.

With the ontology defined, the next architectural choice is the shape of the graph itself and how to extract those entities from raw text.

RDF vs. Property Graphs, and the Three Extraction Modes

Every graph is structured as a collection of (entity, relationship, entity) triplets. But there are two ways to attach data to each entity or relationship instance, known as Resource Description Framework (RDF) and labeled property graphs.

RDF attaches each piece of metadata as another triplet. The graph explodes in size. Property graphs attach metadata as JSON on the entity or relationship.

In practice, GraphRAG and agents use property graphs [3].

Now, during extraction, where we actually map data into our (entity, relationship, entity) triplets, plus their corresponding data, we have three core methods.

Structured extraction is schema-guided. The LLM outputs entities per the Person Ontology.

Semi-structured extraction uses metadata and lineage without an LLM. You parse the email’s links and attachments.

Unstructured extraction uses an LLM without a schema. The LLM invents its own labels. This is useful for discovery, not for grounded retrieval. In other words, we use the LLM to extract triplets without an ontology. Exactly what we said to avoid in the previous section.

Here is the data-source mapping for the Person Ontology of the digital twin:

The Document Ontology can be completely done through semi-structured mechanics, since we already know what document each chunk comes from, the author of each document, and the references between them.

💡 A student asked about open-domain extraction. Exploratory extraction is great early on when you are figuring out what ontology makes sense for your data. You can use zero-shot Named Entity Recognition (NER) models like GLiNER for that exploratory phase [4]. Which you can easily run locally without having powerful inference hardware. Without that discipline, the output becomes unusable noise within tens of documents. A constrained scope lets you swap the frontier model for a small fine-tuned extractor like Gemini Flash Lite, Claude Haiku or even better, use Liquid open-source models fine-tuned on your ontology.

These extraction modes feed directly into a five-component system that turns raw documents into queryable memory.

The Five-Component Architecture

The input consists of heterogeneous documents scattered across multiple silos. The output is a single queryable knowledge graph. The agent can search and write back to it via two tools.

Everything in between is plumbing built to serve that one job.

The data pipeline gathers from URIs, notes, emails and Google Drive. It normalizes everything into a document collection written to a warehouse.

The memory pipeline turns documents into knowledge-graph objects and writes them into the unified memory modeled as a KG.

The KG is the queryable artifact. The agent communicates with the knowledge graph via an MCP server that exposes search and write tools. If you are building in Python, choose FastMCP over the native MCP SDK, as it’s much easier to use and offers a better developer experience.

The search_memory family of tools brings only the slice the agent needs into the context window. The write_memory tools run the same data + memory pipelines on demand on a conversation or URI instead of running them in batch mode [5].

Ultimately, we connect the MCP server to a harness such as Claude Code or Codex, where we inject custom business logic on how the tools should be used through a family of assistant-memory and assistant-learn skills.

For 2-3 hop traversals, Postgres or MongoDB handle documents, vectors, and graph-lookup in a single piece of infrastructure [7].

Reach for Neo4j only when deep traversals or specialized graph algorithms are core to the product [8]. Or a good trade-off is to use it internally just for data exploration. Do not design for Google scale when you are processing thousands of documents.

The memory pipeline sits at the core of this architecture, transforming raw documents into the exact triplets the rest of the system queries.

The Memory Pipeline

The memory pipeline cleans the incoming document.

Next is optional chunking. If you can avoid chunking, avoid it. It introduces problems and is more about RAG-era reflexes than a necessity. You always have to customize the solution based on your data and try to introduce as little complexity as possible.

Next, the graph extractor emits triplets. You should use Pydantic-style schema descriptors so the LLM knows how each field should look.

Normalization is the most important step. You track the evolution of a single entity over time. Do not allow multiple versions of the same person to exist. The system re-uses the same canonical ID across extractions. New metadata and new relationships layer on top [9].

Finally, you embed the relevant fields for semantic search.

Now, let’s look at the core ways of data models you can use to store your KG.

Single Mutable Collection vs. Append-Only Log Data Models

There are two main approaches on how you can model your collections: as an append-only log or as a single mutable collection. Both have their pros and cons.

The append-only log consists of two collections: an append-only log and a queryable materialized view.

The system appends every event to an immutable log. A periodic materialization step squashes all events for the same ID into one canonical record.

You get versioning, temporality, and reversibility for free. You pay in RAM and operational complexity. As RAM is the most scarce and costly piece of hardware for hosting databases, this quickly translates into larger compute costs.

The single mutable collection approach drops the log. Each extraction directly upserts into the queryable collection.

You get simpler ops and real-time visibility, but the temporal audit trail is gone. Pick the single collection if operational simplicity and reduced costs beat time-travel.

Pick the two-collection append-only approach if you genuinely need an audit trail. Append-only collections never delete and never update. The same ID can appear multiple times across extractions, reflecting updates of an entity or relationship instance across the KG.

You can replay history up to a point in time, soft-delete, and revert a bad extraction. Materialization squashes all logs sharing an ID into one canonical entity.

An intuitive way of comparing the two methods is that the single mutable collection option is the same as the materialized view of the append-only option. Thus, one option comes with an append-only log, which comes with versioning and temporality, while the other doesn’t.

How Would This Look Within the Digital Twin?

Each log event lands with an auto-generated ObjectId plus a single chunk_id and source_document_id pinning it to one origin, with no embedding because nothing has been merged yet into the final instance. Materialization groups events by (name, type) for nodes and by the (source, kind, target) triplet for edges, swapping the ObjectId for a deterministic composite ID that is the merge key, unioning every contributing document into a sources array, and embedding each canonical entity once.

Nodes and edges share a single collection, separated only by a kind discriminator. So within our MongoDB implementation, $graphLookup walks source_node_id → target_node_id recursively without joining across collections.

A student asked about community detection and isolated nodes. Once materialization runs, the system computes communities over the canonical node collection. An isolated node is just a singleton community. Filter or keep it based on your use case. Postgres and MongoDB handle hundreds of millions of small records. They can also scale vertically easily through sharding by partitioning on the entity and relationship IDs.

Now, let’s finally understand how we can query the KG and plug it into an agent.

Finally...Let’s Understand the Retrieval Algorithm

During retrieval, we use a hybrid index.

Text search uses exact keywords. Semantic search is meaning-based. Graph search is a multi-hop traversal across the typed edges.

Communities are an optional fourth index for topical clusters.

GraphRAG retrieval is a two-stage move [10].

Stage 1 runs text and semantic search. It merges results with Reciprocal Rank Fusion (RRF). Apply a cutoff to get your entry points [11].

Stage 2 walks 2-3 hops across the typed edges to expand the result set.

During retrieval, GraphRAG’s addition over RAG is this multi-hop step, after the RRF merge, which is standard for most RAG systems.

Still, there are two important details to highlight. There’s bottom-up, which expands entities for depth, while top-down hops across communities for a high-level overview [2]. This translates to a trade-off between context size, latency and performance.

Now, to close the loop, let’s connect everything to an agent.

The Cherry on Top: Agentic GraphRAG

GraphRAG becomes agentic when an agent gets to write to and search the knowledge graph autonomously [5].

The agent dynamically writes queries against the materialized knowledge graph using a family of search_memory tools. The write_memory family of tools runs the data and memory pipelines on the current conversation or any other type of document. These tools are exposed to the agent via the MCP server, implemented in FastMCP.

This differs from the five-component architecture explained earlier: this time, the agent decides when to search/write to memory.

The search tools can directly implement the text + semantic + graph-search algorithm programmatically, or let the agent write the query code on-demand, which gives more flexibility at the cost of potentially less optimal code.

As for the write tools, allowing the agent to ingest the current conversation ensures continual learning by dynamically tracking the user’s preferences, to-dos, experiences and more.

At the moment, harnesses such as Claude Code use the filesystem to implement the memory layer. But as the data grows, gets more complex, or we have to operate under strict cost/latency requirements, we will need more powerful solutions than just hoping the agent will figure it out through progressive disclosure.

What’s Next

In this piece, I presented only the high-level architecture and strategies around GraphRAG.

The issue is that when you start diving into each component, such as normalization, extraction, embedding or data modeling, you will realize that everything is extremely custom to your own data and use case.

This is especially true because GraphRAG is still in its early days, where there is no clear plan of attack.

That’s why I am actively working on a new book on how to implement a personal assistant from scratch (yes, together with Maxime Labonne!), where we will explore building a memory layer stage by stage: RAG, then GraphRAG, with an AI Evals layer on top to measure the actual gain in performance when introducing GraphRAG. As soon as I have more details on this, I will let you know.

But here is what I’m wondering:

Are you using a single database (Postgres / MongoDB) or splitting graph and vector workloads across specialized systems (Neo4j + Pinecone)?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

References

1. Anthropic. (n.d.). Effective Context Engineering for AI Agents. Anthropic. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

2. Larson, J. (2024, April 2). GraphRAG: Unlocking LLM Discovery on Narrative Private Data. Microsoft Research. https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/

3. Negro, A., Kus, V., Futia, G., & Montagna, F. (n.d.). Knowledge Graphs and LLMs in Action. Manning. https://www.manning.com/books/knowledge-graphs-and-llms-in-action

4. Neo4j Graph Data Platform. (n.d.). How Entity Extraction Works. Neo4j Agent Memory. https://neo4j.com/labs/agent-memory/explanation/extraction-pipeline/

5. Monigatti, L. (n.d.). The Evolution From RAG to Agentic RAG to Agent Memory. Leonie Monigatti. https://www.leoniemonigatti.com/blog/from-rag-to-agent-memory.html

6. Govindarajan, V. (n.d.). OpenClaw Architecture - Part 3: Memory and State Ownership. The Agent Stack. https://theagentstack.substack.com/p/openclaw-architecture-part-3-memory

7. Iusztin, P., & Rodrigues, J. (n.d.). How We Killed Our RAG Pipeline.

8. Neo4j Graph Data Platform. (n.d.). Why Neo4j? Graph-Native Memory Architecture. Neo4j Agent Memory. https://neo4j.com/labs/agent-memory/explanation/graph-architecture/

9. Neo4j Graph Data Platform. (n.d.). Entity Resolution and Deduplication. Neo4j Agent Memory. https://neo4j.com/labs/agent-memory/explanation/resolution-deduplication/

10. Hedden, S. (n.d.). How to Build a Graph RAG App. Towards Data Science. https://towardsdatascience.com/how-to-build-a-graph-rag-app-b323fc33ba06/

11. Arancio, J. (n.d.). Comment on Hybrid RRF Retrieval Pipeline. Substack. https://substack.com/@jeremyarancio/note/c-205294494

12. Liu, J. (2025, May 19). There Are Only 6 RAG Evals. jxnl. https://jxnl.co/writing/2025/05/19/there-are-only-6-rag-evals/

13. Zhang, B. (2026, January 22). Scaling PostgreSQL to Power 800 Million ChatGPT Users. OpenAI. https://openai.com/index/scaling-postgresql/

14. Govindarajan, V. (n.d.). OpenClaw Architecture - Part 2: Concurrency, Isolation, and the Invariants That Keep Agents Sane. The Agent Stack. https://theagentstack.substack.com/p/openclaw-architecture-part-2-concurrency

Images

If not otherwise stated, all images are created by the author.

Most AI demos work. Most AI products don’t. This series is a collection of interviews with engineers who shipped AI agents to production, covering the stacks they chose, the architectures they regretted, and what actually held up at 3 am.

This is an interview with Michael Maximilien, former CTO and Distinguished Engineer at IBM and Chairperson of the Board of the NodeJS Foundation. Now, the founder and CEO of ClawMax.ai, an AI agent orchestration platform powered by OpenClaw and the creator of weave-cli, an open-source tool for shipping Retrieval-Augmented Generation (RAG) systems.

Watch our full interview on YouTube ↓

Michael Maximilien spent a year building RAG systems for customer after customer. Every new project required navigating dozens of moving parts. He had to pick a vector database, select an embedding model, chunk the data, ingest it, search it, and iterate.

“I was doing this a lot and I wasn’t getting the results I wanted.” — Max

The failures were concrete. Halfway through an ingestion run, Milvus would run out of memory. Two collections made it in. The third was broken. Without a checkpoint or resume function, he had to recompute everything from scratch.

“The experiment doesn’t just run, it fails. You have to be able to pick up from the failure.” — Max

Another failure mode involved manually comparing Weaviate against Milvus. One configuration typo could lead to drawing the wrong conclusion.

“You might end up thinking Weaviate is better than Milvus when actually your comparison was wrong.” — Max

This manual flywheel stole time from actually helping his customers ship their products. He burned days on reset and re-ingest cycles that failed halfway. Worse, he produced results he could not trust.

Most teams treat RAG as a simple setup task. They picked a vector database because it trended online. They pick an embedding model because OpenAI is the safe default.

They guess a chunking strategy, guess the top-K retrieval parameters, and ship it. Then they spend the next six months vibe-checking the system.

Users complain. The team swaps a configuration knob. Nobody knows if it actually helped because nothing was measured.

“There’s a lot of steps.” — Max

You lose the working system you thought you had. You burn weeks debugging silent ingestion failures because no trace exists.

Customer trust evaporates when the same question gets three different answers across releases.

Max took the opposite bet. He built Weave CLI: a unified command-line tool for RAG over eleven vector databases.

It features first-class observability implemented with Opik, an open-source evaluation and optimization tool, baked in from the first commit. You can try out their managed platform for free here for 25k spans/month.

By the end of this case study, you will understand how to unify your RAG stack so that switching a database, an embedding model, or an agent is merely a config change. You will learn how to measure everything, so every switch is tracked, evaluated and compared. Ultimately, you will learn how to benchmark your solution against multiple parameters to find the best configuration for your problem.

“There’s no one solution. You iterate and evaluate.” — Max

But first, let’s understand what Weave CLI is and how it works.

Understanding the System Architecture of Weave CLI

Weave CLI wraps 11 vector databases behind a single interface. From the outside, it looks and feels like any other RAG system. On the ingestion side, it populates the chosen vector database with chunks, metadata, and embeddings. On the query side, it takes natural-language questions and returns top-k ranked chunks that an agent can use to create an answer with citations.

What makes Weave CLI special is that everything is swappable via a configuration file: the vector database, the embedding model, the chunking strategy, the query agent, the RAG agent that interprets the chunks and so on. With the goal of making it very easy for you to benchmark, iterate on and improve your RAG solution.

Weave CLI is composed of seven core components, each swappable by configuration.

The user-facing component is the Cobra-based CLI and the interactive REPL. Weave stack sits underneath as the deployment layer. It brings the whole system, the databases, up or down with a local Docker/Podman Compose fallback.

Behind that surface sits the intelligence layer. Ten built-in agents share an AgentChain sequencer. Agents are used both within the CLI and during ingestion. Weave CLI supports RAG, QA and summarization agents, but what’s more interesting is during ingestion. For example, you describe your data, the SchemaAgent proposes a collection schema and a vector-database fit, the ChunkingAgent recommends a chunking strategy and an embedding provider is picked to match. The Executor drives a seven-step orchestration covering query analysis, planning, user confirmation, execution, reporting, display, and evaluation metrics.

The data layer is built around the VectorDBClient interface in src/pkg/vectordb/interfaces.go. It is cleanly split into four sub-interfaces: CollectionOperations, DocumentOperations, QueryOperations, and SchemaOperations. A package-level factory registry in factory.go registers all 11 adapter sub-packages using the ports-and-adapters pattern.

Still... There is a trade-off to this design. A unified interface is a lowest-common-denominator by construction, so if you need PGVector’s transactional semantics or Neo4j’s native graph traversal as first-class features, a unified adapter costs you that expressiveness.

On top of the vector-database layer sit five embedding providers: OpenAI, sentence-transformers, Ollama, Cohere, and Voyage. The ingestion pipeline runs alongside these, handling file scanning, processing, and batching.

As of April 2026, Max has strong views on which vector database to choose. Weaviate is his default for cloud deployments. Pinecone is the pick for hosted solutions. OpenSearch covers self-hosted cloud. Milvus handles both local and cloud. Qdrant is his go-to for local use because its Rust implementation is low-memory and fast.

On top of the agent layer, we have the observability layer implemented using Opik and OpenTelemetry, along with an evaluation harness with four LLM judges. The evaluation harness is itself pluggable between a local evaluator and Opik.

The configuration is the source of truth for the whole stack. A config.yaml file holds the non-secret details of the vector database, agent, embedding model, and LLM, while secrets are loaded from a .env file. Check all the configs here.

Let us trace a query through the system end-to-end using a concrete example: a user asks about Leica Noctilux lens auctions. The flow unfolds across nine hops, each an Opik span. First, the user submits the natural-language query to the REPL, which immediately starts monitoring the trace with Opik.

The QueryAgent then validates and classifies the intent, passing control to the PlanningAgent to generate an execution plan. Next, the VectorDB adapter performs a semantic search to retrieve relevant documents. The ContextBuilder filters, deduplicates, and sorts these results before handing them to the RAGAgent, which generates the final answer with citations. Finally, the REPL ends the Opik trace, in which every step emits a span containing details such as costs, latency and input/output.

Why Building in Go and Not TypeScript or Python

Most popular agentic CLIs / REPLs, such as Claude Code or OpenCode, are built in TypeScript. But Max, as a former Node.js board chairperson, strongly suggests just going with Go or Rust if memory constraints are a concern.

Why? Because Go apps ship as single binaries. Simple. Beautiful. It runs everywhere.

On-premise customers cannot rely on npm, uv, or JVM registries being reachable inside their own networks, and dependency pinning does not fix network isolation. A single compiled binary sidesteps the entire problem. Go’s track record as the language of Kubernetes is the existing proof that this trade-off works for infrastructure tooling, which is exactly the category Weave CLI sits in. Max himself spent 10 years writing Go on those systems.

Max’s second argument is that language choice matters less than it used to, because AI coding assistants lower the learning-curve barrier across the board.

“Most people don’t write code anymore.” — Max

Still... Max’s newer project (ClawMax.ai) is mostly in TypeScript because it is the best tool for the job, not because he switched allegiances.

“The stack decision has to be what your system wants, not what the herd is doing.” — Max

Next, we zoom into the layers doing the heavy lifting, the ingestion pipeline and the unified VDB interface.

Supporting 11 Vector Databases

The vector database layer is where the ingestion pipeline meets the unified VDB interface. To see how they work together, we’ll trace Max’s Leica Noctilux auction catalog through the system one step at a time.

Each document in the catalog is a single lens listing. It contains a photo of the Noctilux, a short caption with the model number and condition, a price, and a few lines of provenance. The text is sparse. Most of the signal sits in the image itself, and the caption is just enough to disambiguate one Noctilux from another. That sparseness drives a multi-modal ingestion decision up front. The image and the surrounding caption are embedded into two separate collections, one keyed on image vectors and one on caption text vectors. At query time, the auction agent fans out to both collections and merges the results through the ContextBuilder.

Before the actual ingestion, we run a FileScanner that walks the 426 listing files on disk, applying glob matching, exclusion filters, and SHA256 deduplication (src/pkg/pipeline/scanner.go). Re-running ingestion on the same directory skips unchanged documents, making this step fully idempotent and computationally cheap.

The DocumentProcessor extracts text and images from each listing (src/pkg/pipeline/processor.go). For the Leica dataset, the PDF extractor pulls the caption text, and OCR runs on the lens photo to catch any model number printed on the barrel. This step is idempotent but computationally expensive due to PDF parsing and OCR, and it fails if the document format is unsupported. Next, the ChunkingAgent dynamically selects the best chunking strategy for each document.

Next, we move to embedding (src/pkg/embeddings/model_registry.go). In the Leica flow, caption text flows through the text embedder, and image descriptors flow through a separate image embedding model. Raw images larger than per-backend limits (Milvus caps fields at 65KB) get offloaded to S3/MinIO, leaving only a URL in the VDB payload. The default option is to use OpenAI’s embedding model, which is highly expensive in compute and API costs and can fail if you hit rate limits. When scaling, you can use open-source embeddings via Ollama. They run locally with no API key.

The BatchWriter processes documents with durability, such as checkpoint and resume functionality. For example, when ingesting data at scale, you often have network I/O failures or database connection drops. Through checkpointing, we ensure the state is idempotent. Batch checkpointing is the difference between a short retry and a multi-hour rebuild.

“You have to recompute everything from scratch, which is crazy.” — Max

The VectorDBClient Interface sits at the core of the adapter pattern (src/pkg/vectordb/interfaces.go) used to support the 11 databases. The project started with Weaviate. Milvus was surprisingly similar. Qdrant was also very similar. MongoDB was a different beast, but the interface still fit.

“The biggest surprise was PGVector.” — Max

PGVector is the most incompatible on paper. Postgres is a relational database with its own migrations. Yet the unified interface fits.

The pipeline ends at any of the eleven vector databases (src/pkg/vectordb/factory.go), emitting a final vectordb.adapter span. The 426 Leica listings are split into roughly 426 caption vectors in one collection and 426 image vectors in a parallel collection, both sharing listing IDs as the cross-reference key.

These steps cover every component any production ingestion pipeline needs, and Weave CLI ensures each one is swappable by configuration (src/pkg/stack/ingest.go): the FileScanner, the DocumentProcessor, the ChunkingAgent, the embedding provider, the BatchWriter, the VectorDBClient interface, and the concrete VDB adapter.

During retrieval, when a user asks weave query "summarise the 2024 auction catalogue", the QueryAgent classifies the intent, the PlanningAgent decides to hit both Leica collections, and the VectorDB adapter runs a semantic search on each. The ContextBuilder then merges the image-collection hits with the caption-collection hits, deduplicates by listing ID, sorts by relevance score, and extracts content in priority order (caption text first, image metadata second, URL fallback last) into a single prompt for the RAGAgent.

The ingestion pipeline and VDB interface are the skeleton of Weave CLI. The agent layer is what makes it feel like Claude Code for vector databases.

Zooming into the REPL

Weave CLI provides a Claude-Code-like experience for vector databases, which, at its core, is a Read-Eval-Print Loop (REPL) environment hooked up to multiple agents.

Weave CLI ships with 12 built-in agents that you configure via YAML. Three of them are user-facing:

1. Precise QA — asks a question and answers it, and says it cannot answer when it lacks information. Zero hallucination tolerance.

2. RAG — finds the closest chunks and generates an answer over them. This is the default.

3. Summarize — produces a short summary of retrieved chunks.

💡 The beauty is that you can add or modify them as you please.

The next eight agents power the Claude-Code-like orchestration loop: the QueryAgent for intent classification, the PlanningAgent for the execution plan, the WeaveAgent for tool execution with retries, the BashAgent for safe execution, the RAGAgent that the RAG persona dispatches to, the OutputAgent to format progress, the ReportAgent to generate operation reports, and the EvalAgent to track metrics.

The final two are domain helpers used during ingestion: the ChunkingAgent and the SchemaAgent.

Similar to the vector database layer, all the agents implement the same interface:

Let’s tie everything together. When you ask a query, the QueryAgent classifies intent and acts as a router. The PlanningAgent generates a plan of CLI commands. The BashAgent executes them and pipes the output through a command-line JSON processor for filtering. The OutputAgent formats the result. This is the Claude-Code-like loop in action.

The cherry on top is that the Weave CLI capabilities are also exposed as a Model Context Protocol (MCP) server. Thus, instead of using the Weave CLI directly, you can leverage its full functionality through your harness of choice (Claude Code, Codex, etc.).

Twelve agents, eleven databases, five embedding providers, and multiple chunking strategies create a lot of surface area. Opik is what makes the whole thing observable when something breaks.

Monitoring the System

With so many moving parts, you need to know the system is working. Opik is how Weave CLI answers that question: it traces every LLM call, every agent step, and every database write as an OpenTelemetry span.

“Using Opik to tell me how many LLM calls, tokens, and cost per query.” — Max

During development, Max tracked a bug in which documents appeared to be ingested but were never persisted to Milvus. The Opik trace waterfall showed the database flush operations were silently timing out.

💡 If you want to try it out, you can create an account for free on Opik’s managed platform here for 25k spans/month.

The fix was adding dedicated timeout contexts per collection. Without the trace, this would have been a multi-day hunt through logs.

The integration provides cost and latency visibility per trace. You see tokens and dollars per query without writing custom logging. It provides a latency breakdown.

Finally, it provides error visibility to make silent failures loud.

How hard was it to integrate Opik into Weave CLI?

“It’s a very straightforward integration — I pass all queries to the LLMs through Opik via OpenTelemetry, and then I query Opik to aggregate cost from the start of the command to the end.” — Max

Every step in the ingestion and retrieval data flows emits a span (src/pkg/llm/opik.go), which are aggregated under traces containing all the steps between a user request/response.

It includes the query, the LLM reasoning, the tool calls, and the final response. The executor initializes Opik tracing here (src/pkg/executor/executor.go).

Monitoring helps you debug your system. Evaluation moves everything forward, allowing you to quantify your application’s performance.

Evaluating the Default Setup

How do you know your agent is actually better after you swap an embedding model, a vector database or your chunking strategy? You need a good evaluation practice.

“My customers always have five or six questions they ask every release to sanity-check the system. They know what to expect. So I took their QA questions and made them the baseline eval dataset.” — Max

Evaluation datasets come from real user behavior anchored in your business use case, not from standardized, generic benchmarks. If you do not have users yet, you should compile a small set of sanity questions a domain expert would actually ask.

How does this work in Weave CLI?

You start by defining an evaluation dataset in YAML format. This includes the query, expected answer, expected citations, and a minimum relevance score.

Here is the full baseline.yaml file. Or this is how it looks in Opik:

Then you pick an evaluator harness that includes a set of metrics to evaluate against. This harness is itself pluggable: you pick between a local evaluator and Opik (src/pkg/evaluation/provider.go).

We use two families of evaluators. Rule-based evaluators use regular expressions, exact matches, and citation presence (src/pkg/evaluation/custom_evaluator.go) to compute metrics such as CitationMatching for the RAG agent.

They are fast, deterministic, and free. You use them for structural checks.

The second family uses an LLM as a judge. Weave CLI ships four of these judges (src/pkg/evaluation/provider_opik.go). They evaluate Accuracy, Faithfulness, Hallucination, and Context Relevance.

They are slower and cost tokens. You use them for semantic quality.

“The hallucination, citation, and accuracy metrics are all from Opik’s library — I ported them to Golang.” — Max

Then, you run the evaluation command against a chosen agent. Finally, you compare the result of the experiment with the previous run. Each pair of agent and dataset is one experiment (src/cmd/eval/run.go).

The --use-opik flag ships every trace and evaluation result to Opik (src/pkg/evaluation/runner.go). Once in Opik, you get dataset management and experiment comparison.

Scoring every run forces a decision on which agent to ship. Benchmarking on top of your custom datasets provides a structured way to choose a parameter, such as your chunking strategy or top k results, without guessing.

Benchmarking and Optimizing the System

An experiment is a single parameterized run over an agent, dataset, embedding, chunking strategy, database, and judge. A benchmark is a structured set of experiments.

You hold most variables constant to isolate the effect of one. Benchmarking is how you turn random runs into a parameter- and prompt-search problem. This is often known as the optimization flywheel.

“That’s the reason I created Weave CLI. Because this is tedious, but also error-prone.” — Max

Every benchmark is one configuration typo away from drawing the wrong conclusion. Disciplined benchmarking catches that error.

Experiment metadata guarantees reproducibility. Every experiment records the database, embedding model, chunking strategy, dataset, and everything else required to reproduce it. That’s usually the whole config.

Opik tracks this out of the box. Without it, a benchmark from four weeks ago is useless.

When working on RAG systems, the optimization flywheel involves resetting the database, re-ingesting data with new parameters, re-evaluating, and comparing on your metrics of choice.

“Benchmark is comparing multiple agents side by side. Same dataset, different agents — and each (agent, dataset) combination is its own experiment, you can compare later with its metadata.” — Max

You fix a baseline dataset and hold it constant. You vary one axis, typically the agent. You score against multiple metrics.

Each pair of agent and dataset is one Opik experiment. You compare them side-by-side to spot regressions and unexpected wins.

You trigger this loop via the command line with weave eval run --dataset baseline --agents precise-qa,rag,summarize --use-opik. Every subsequent benchmark streams into the same Opik project.

Max ran this loop for his Leica auction customer. He held the dataset and agent constant.

He varied only the embedding provider. He tested OpenAI against sentence-transformers. The open-source model won on quality by 11 percent.

It was 240 times faster for re-embedding. The vectors were 50 percent smaller, and the cost was zero.

This is a counterintuitive outcome. Without a structured benchmark, Max would have defaulted to OpenAI and been wrong.

How to Keep the Flywheel Under Control?

This optimization process involves running your ingestion and retrieval hundreds of times. Which can get costly fast. Super fast. The ingestion checkpointing makes it affordable.

Still, you should optimize your system in order of cheapest-to-change, biggest-win-first [8]. First, tune retrieval parameters like top-K. They are free to change and often provide the biggest wins.

Second, tune the embedding model. It is the cheapest component to swap and has a huge impact. Third, tune the chunking strategy. It requires re-ingestion but offers moderate quality gains.

Finally, tune the vector database. It has the highest switching cost and usually the smallest difference in quality.

The optimization flywheel effectively isolates variables, but it remains a manual process today.

The good news is that Weave CLI is heading toward full automated hyperparameter optimization across databases, embeddings, and chunking strategies. Just imagine. You will launch it before the weekend, and it will return on Monday with the best configuration for your dataset.

You can reproduce this benchmark step by step on your own stack by following this doc.

Watch our full interview on YouTube for all the 3am stories ↓

Final Thoughts

Looking back, what was the hardest thing to implement, and what surprised you the most while building weave-cli? — Paul

The hardest part was designing a unified VectorDBClient that felt natural across 11 providers with wildly different APIs. The adapter pattern was the insight that made it work.

The biggest surprise was benchmarking OSS embeddings against OpenAI on the client’s data and finding them 11% higher quality, 240x faster, and free. A call we’d never have made without evals in place.

If you had to rebuild Weave CLI from scratch, at what point would you introduce monitoring and evaluation? Would you do it earlier, later, or at the same time? — Paul

I’d introduce monitoring from day one. Having Opik traces during the early vector DB work would have immediately surfaced issues such as the silent Milvus persistence failures, which we debugged manually. As for evals, I’d keep at the same stage (after the core RAG pipeline was functional), but I’d design the harness interface up front for citation tracking and confidence scoring.

Opik was easy to integrate and was key to getting the client dashboard working, since I could just run experiments and use evaluations and tracing to decide on the best options for the client.

Now, your next practical step is to experiment with Weave CLI on a real problem. Point it at 100 documents you want to do RAG on, ingest everything into two collections with two different embedding providers, and run the benchmark against the baseline evaluation dataset.

You can follow the step-by-step tutorial from here

But here is what I’m wondering:

While building your latest RAG system, what was your strategy to find the right parameters, such as the embedding model, chunking or retrieval strategies?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

Thanks again to Opik for sponsoring this case study and keeping it free!

If you want to monitor, evaluate and optimize your AI workflows and agents:

Try Opik for free

References

1. Chroma. (n.d.). Evaluating Chunking Strategies for Retrieval. Chroma. https://research.trychroma.com/evaluating-chunking

2. OpenTelemetry. (n.d.). Traces & Spans specification. OpenTelemetry. https://opentelemetry.io/docs/concepts/signals/traces/

3. Husain, H. (n.d.). Creating a LLM-as-a-Judge That Drives Business Results. Hamel Husain. https://hamel.dev/blog/posts/llm-judge/

4. Husain, H. (n.d.). Escaping POC Purgatory: Evaluation-Driven Development for AI. Hamel Husain. https://hamel.dev/blog/posts/evals/

5. Liu, J. (2025, May 19). There Are Only 6 RAG Evals. Jason Liu. https://jxnl.co/writing/2025/05/19/there-are-only-6-rag-evals/

6. Comet. (n.d.). Opik — LLM observability & evaluation platform. GitHub. https://github.com/comet-ml/opik

7. Yan, E. (2024, August 18). Evaluating the Effectiveness of LLM Evaluators (LLM-as-Judge). Eugene Yan. https://eugeneyan.com/writing/llm-evaluators/

8. Liu, J. (2024, February 28). Levels of Complexity: RAG Applications. Jason Liu. https://jxnl.co/writing/2024/02/28/levels-of-complexity-rag-applications/

Images

If not otherwise stated, all images are created by the author.

I eventually collapsed the system into two agents: a writer and a reviewer operating in a loop. The simpler version performed better. The model retained the full context, and verification became a simple review step rather than a massive orchestration problem.

Most AI teams hit this exact wall. Developers spend more time babysitting AI than engineering, copying error logs and re-prompting models. The real bottleneck is the human.

Three failure modes explain why.

First, context rot. In long AI conversations, the context window becomes a junk drawer. Every failed attempt piles up until the sliding window drops the original specification. The model slides into a “dumb zone” where it hallucinates and forgets its goals. Traditional fixes like summarizing break down over dozens of reasoning rounds.

Second, premature exit. AI agents declare victory too early. Anthropic’s research notes that agents usually look around, see that progress has been made, and declare the job done [1]. Standard ReAct loops inherit the flaw.

Third, single-pass fragility. One prompt, one context, one shot. When it fails, the failure is chaotic. Jumping to multi-agent orchestration introduces distributed systems nightmares.

Ralph loops break the cycle by making “try again with fresh eyes” the default. Named after Ralph Wiggum from The Simpsons, the pattern wipes the conversation, reloads the full specification fresh each iteration, and uses the filesystem and git as the memory layer.

They remove the AI’s ability to grade its own work, using objective signals such as passing tests or linters to call the job done. Boris Cherny, creator of Claude Code, states that giving Claude a way to verify its work increases quality two to three times [2].

One model. One loop. One verification signal. Failure becomes predictable, the loop catches errors and re-prompts automatically, creating a relatively deterministic feedback loop that will 10x the quality of the agent.

Now, let’s look at what Ralph loops are and when you can actually use them in practice.

What Ralph Loops Are and How They Work

Geoffrey Huntley named the pattern after Ralph Wiggum from The Simpsons, noting the character tries the same thing over and over until it works. Huntley’s motto captures the philosophy: the technique is deterministically bad in an undeterministic world. The simplest implementation is a bash while-true loop that pipes a prompt file into the agent forever, acting as a continuous harness pattern [3], [4].

As Einstein reportedly said: “Insanity is doing the same thing over and over again and expecting different results.” Well... I am sure he didn’t predict the rise of Claude Code, because that’s exactly what Ralph loops are all about.

Models are stochastic, strong at reading large contexts but imperfect on first pass. Re-running the same instruction forces self-review. The first iteration produces good but flawed output.

During the second pass, the model spots what it missed and refactors. The third iteration handles cleanup. Huntley delivered a minimum viable product quoted at 50,000 for just 297 in tokens using a single Ralph loop: a 170x cost reduction over the human estimate [3].

You can run these loops in two modes. Shared context keeps the session alive for explicit self-review. Fresh context starts a new session each iteration, removing confirmation bias.

The model sees only the repository and the skill file.

Nowadays, it’s common to replace brittle n8n workflows with a single Claude Code skill. This becomes even more powerful when running in a Ralph loop, especially because at the end of each run, you can take the signal and tell the model to update the skill with anything it should have done differently.

The skill evolves and quality improves automatically [5].

This self-improving mechanism reduces manual prompt tuning when applied to specific, repetitive engineering tasks.

Three Real-World Use Cases

In practice, Ralph loops don’t have a clear implementation pattern. They are more of an intuitive strategy you can get creative with. Thus, you have multiple ways of implementing them.

From my experience with Claude, you have three options for running Ralph loops, from highest abstraction to lowest:

• /ralph-loop plugin — the fastest path. Install it, run /ralph-loop in your session, and it manages the cycle for you.

• /loop command — Claude Code’s built-in scheduler. /loop every 1 minute /your-skill fires the skill on a schedule [6].

• while true bash loop — the most primitive form. A one-liner that pipes a prompt file into the agent and restarts it forever.

Because Claude Code keeps state through the files it’s working on, it retains context from the failed attempt and reads its own git diffs. Each iteration learns from the last.

Implementing a ticket backlog with test-driven development

You can set up a ticket folder with numbered text files. Run a while-true loop that tells Claude to implement the next most important ticket using Test-Driven Development (TDD). The model writes tests first, writes the code, commits the changes, and moves on.

Claude reads all tickets, skips completed ones, picks the next priority, implements it, marks it done, and commits. No dependency graph is needed because the model decides the ordering on the fly. One dumb loop acts like a relentless single-threaded engineer working through the backlog.

For example, set up a doc/tickets folder with numbered tickets (001, 002, 003...). Each describes a feature or fix. Then run:

while true; do
  claude "implement the next most important ticket using TDD principles from doc/tickets. commit when done"
done

Or use Claude Code’s built-in loop:

/loop every 1 minute
build the next ticket from doc/tickets using TDD, run tests, commit when done

Adding test coverage

You can set a concrete goal to raise coverage from 16 percent to 95 percent. The loop reads coverage metrics, writes tests for uncovered functions, runs the suite, identifies gaps, and iterates.

The coverage report provides the objective backpressure. The loop does not stop until the numbers validate success. Each iteration chips away at untested code paths until the threshold is met.

The implementation is as easy as:

while true; do
  claude "analyze coverage gaps, write tests for uncovered functions, run the test suite, fix failures. stop when coverage exceeds 95%"
done

Framework and dependency migrations

Migrations require crisp completion criteria. Upgrading React v16 to v19, Next.js 14 to 15, or migrating Jest to Vitest demands a clean build and passing tests. The agent swaps syntax, updates dependencies, and runs build commands.

It uses compiler errors and failing tests as feedback. Each cycle fixes a batch of errors until the toolchain confirms the code is clean. Deterministic verification signals make framework migrations the perfect Ralph loop candidate.

These are three concrete starting points. Before you wire the first one up, there is one honest limit you should know.

What’s Next

Ralph loops are the starting point. Once comfortable, add self-improving skills that update their instructions after each run, wire stop hooks for objective quality gates to avoid infinite loops, and connect to external systems like Linear or GitHub Issues so the loop reacts to new work automatically.

The pattern scales further than it looks. OpenAI’s Codex team shipped one million lines of code across 1,500 pull requests with zero human-written code using what they call a “Ralph Wiggum Loop” [7].

These loops are safe when repo-contained and the toolchain acts as the judge. They get dangerous with irreversible side effects outside the repo. Alexey Grigorev learned this when a Claude Code agent ran terraform destroy on DataTalks.Club’s production infrastructure, wiping the database, VPC, and all automated snapshots — two and a half years of data gone in one iteration. If your loop can destroy shared state, review every plan manually [8].

What is the first piece of work in your repo you would trust a Ralph loop with? You could choose a TDD backlog, a coverage ramp, a framework migration or what else?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

References

1. Anthropic. (n.d.). Effective Harnesses for Long-Running Agents. Anthropic. https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents

2. Cherny, B. (n.d.). I’m Boris and I Created Claude Code. X. https://x.com/bcherny/status/2007179832300581177

1. Huntley, G. (n.d.). Ralph Wiggum as a “software engineer”. Geoffrey Huntley. https://ghuntley.com/ralph/

2. LangChain. (n.d.). The Anatomy of an Agent Harness. LangChain Blog. https://blog.langchain.com/the-anatomy-of-an-agent-harness/

3. Parsons, C. (n.d.). Ralph Loops: Build Dumb AI Loops That Ship. AI Engineer. https://read.readwise.io/read/01kp5bgy8b07y256ythvkz2tt7

4. Anthropic. (n.d.). Claude Code: Best Practices for Agentic Coding. Anthropic. https://www.anthropic.com/engineering/claude-code-best-practices

5. Lopopolo, R. (n.d.). Harness engineering: leveraging Codex in an agent-first world. OpenAI. https://openai.com/index/harness-engineering-codex/

6. Grigorev, A. (n.d.). How I Dropped Our Production Database and Now Pay 10% More for AWS. Alexey Grigorev. https://alexeygrigorev.com/posts/dropped-production-database/

Images

If not otherwise stated, all images are created by the author.

I keep my notes in Obsidian, my reading in Readwise, and my topical research in NotebookLM. Each tool is excellent in isolation, but no AI can reach across all three.

Whenever I reach for a general-purpose deep-research tool like Perplexity or Gemini Deep Research, it just searches the public web. Every user gets the exact same sources, and the resulting article reads like everyone else’s. What I actually want to research is my own curated thinking.

I want to leverage the books I highlighted, the notes I wrote, and the transcripts I dumped into NotebookLM. That is the edge. That is the signal nobody else has.

To solve this, I built a deep research agent as three Claude Code skills. The /research_create, /research_search, and /research_distill skills run on top of my private data via the obsidian, readwise, and nlm command-line interfaces (CLIs).

The system uses multi-round query expansion with gap analysis between rounds. It outputs a memory/ folder with an index.yaml file that acts as a progressive-disclosure wiki over the source files. We also apply post-processing, including deduplication and re-ranking, to keep the result focused.

There is no vector database and no Retrieval-Augmented Generation (RAG) pipeline. We use the filesystem as state and Markdown, YAML, and JSON as the wire format. If you already keep notes in Obsidian, articles in Readwise, or research in NotebookLM, this is for you.

By the end of this article, you will know exactly how it works, see it run on this very article, and have a blueprint to build your own.

Here is the system at a glance. We will look at the three skills, three CLI adapters, and one memory folder, before we open the heaviest skill in the next section.

Your Path to Agentic AI Engineering for Production (Product)

The three-skill + memory-folder pattern in this article is one slice of harness engineering. If you want to master the rest, such as orchestration, context engineering, evals, and production deployment, check out my Agentic AI Engineering course, built with Towards AI.

34 lessons. Three end-to-end portfolio projects. A certificate. And a Discord community with direct access to industry experts and me.

Rated 5/5 by 300+ students. The first 6 lessons are free:

Start here

Three Skills, Three CLIs, One Memory Folder

The system relies on three distinct skills. First, /research_create builds a memory/ folder from scratch for a given topic or brain dump. Second, /research_search handles the read side, letting any future agent query an existing memory/ folder via index.yaml with progressive disclosure.

Third, /research_distill takes a finished piece of content and extracts only the sources that were actually used into a single portable research.md appendix.

The memory/ folder is built around index.yaml. It holds metadata per source, including uri_highlights, uri_full, original_path, and origin. The LLM reads the index first, then picks three to five relevant files based on summaries and reads those directly.

There are no embeddings, no chunking, and no vector store to maintain, ensuring references stay perfectly traceable. Like OpenClaw, we treat memory as plain Markdown in the agent workspace, where files are the source of truth and the model only remembers what gets written to disk [1].

The Obsidian, Readwise, and NotebookLM files act as the raw, immutable data. We touch them manually as humans, never through this pipeline. On top of that, /research_create produces a local actionable knowledge base for a specific scope, resulting in an ephemeral memory/ folder per topic.

This separation allows the same raw data to feed many different research projects without contamination. The key invariant of this architecture is that the orchestrator never loads source files. Researcher subagents touch the raw files, while the orchestrator only ever sees structured JSON summaries flowing between steps.

We chose CLIs over Model Context Protocol (MCP) servers for three reasons. First, token economics. A skill enters Claude Code’s context at boot at ~100 tokens of metadata, and the body loads only when invoked.

By comparison, Notion’s MCP server dumps roughly 20,000 tokens of self-documenting tools at startup whether you use them or not. That is roughly 200× less context before you have done anything [2].

Second, CLIs compose with bash. The orchestrator can pipe results through tools like jq or redirect output straight to a file, whereas MCP tool calls must round-trip through the LLM.

Third, Markdown is the native language of LLMs. As Simon Willison argues, Markdown with YAML frontmatter is more in the spirit of LLMs than MCP, because you put text in the context and let the LLM pick [3].

That is the whole architecture. Now let’s open up the heaviest of the three skills, /research_create, and watch the multi-round research loop in detail, where the orchestrator-never-loads invariant earns its keep.

How /research_create Works

The process starts with a brain dump from the user, which can include text, URLs, or local file paths. During the deep research, you confirm three configuration knobs in one prompt: the number of rounds, queries per round, and a topic slug. Seed URIs from the brain dump always land in the output with a relevance score of 1.0, bypassing reranking because they are your explicit picks.

The orchestrator generates queries and dispatches one researcher subagent per query in parallel. Each researcher runs platform-specific searches. For Readwise, this means querying the library, feed, highlights, and document notes. For Obsidian, it means querying the local vault files. For NotebookLM, it means querying the projects and their associated sources and notes.

For Obsidian, we found that using its CLI — which leverages its index — is 10× more efficient than letting the LLM roam around your vault.

The subagent does its own within-agent deduplication by original path. It captures metadata while files are open. It also caps output at a top-15 limit of unique findings.

Between rounds, a gap_analyzer subagent reads the deduplicated findings via jq without full reads. It flags thin or missing themes against the initial key themes and emits the next round’s queries. After all rounds, a reranker subagent scores every candidate between 0.0 and 1.0 using the cheapest sufficient signal. It checks metadata first, then reads the head and tail of the doc, and uses full reads only as a last resort.

Finally, a builder subagent invokes a Python script to emit the YAML deterministically, placing seeds first, then descending by score.

We use this shape because context isolation is our central design choice. Every step that touches real source content runs in an isolated subagent with its own context window. The orchestrator only sees the compacted metadata of each file, while moving the actual file using mv bash commands into the memory folder.

The index.yaml file holds pointers and metadata for every file in the wiki. The orchestrator holds pointers, while subagents hold content. Geoffrey Huntley, creator of Ralph Loops, states that your primary context window should operate as a scheduler, scheduling other subagents to perform expensive allocation-type work [4].

Subagents compress tens of thousands of input tokens into 1,000–2,000 output tokens before handing back to the orchestrator. That compression ratio is the whole point. The researcher subagents read deeply, and the orchestrator stays light.

Once the memory/ folder exists, anyone can read it without loading source files. We use the /research_search skill to query this index.

How /research_search Works

The /research_search skill handles the read side of the system. Any agent can be handed a memory/ folder and query it without loading source files into context. The skill encodes the protocol once so future agents do not have to re-derive it.

The system uses three layers of detail. Layer 1 is the summary field in index.yaml. It contains two to three sentences per source and is always loaded as part of the index. It is enough to answer what you have on a topic or build a table of contents.

Layer 2 is the key-highlights file, which holds the condensed topics of a file. This is extremely powerful when using reader tools such as Readwise, as these highlights are made manually by you, the reader, consisting of huge signal. Thus, not every source has this layer. It’s better not to have it at all than to have an LLM extract it.

Layer 3 is the uri_full file, representing the complete original document. You read it only when key highlights are insufficient or inexistent.

Anthropic notes that models are great at navigating filesystems, and presenting tools as code on a filesystem allows models to read tool definitions on-demand, rather than reading them all up-front [5]. That maps exactly onto index.yaml plus lazy key-highlights loading.

Intuitively, the index.yaml file gives us progressive disclosure — the same pattern used inside skills — so the agent can choose from many options without drowning in information [6].

The agent slices index.yaml by origin, location, relevance-score threshold, tags, author, publication, date range, or NotebookLM notebook.

The most beautiful part? Because index.yaml is structured data, the agent writes code on top of it. It uses jq filters, Python sorts, and awk projections.

LlamaIndex’s head-to-head benchmark proves this scales. A filesystem-explorer agent beat a hybrid vector RAG pipeline on correctness (8.4 vs 6.4) and relevance (9.6 vs 8.0) at a sub-60 document scale, precisely because the LLM saw whole files instead of chunks [7].

Portability comes for free. Hand the self-contained memory/ folder to any agent, and they get up to speed instantly. Search lets agents find what is there. But once you have drafted an article, you also need to know what part of the wiki you actually used within your piece. That is /research_distill.

How /research_distill Works

Given any piece of content and the memory/ folder used during writing, the skill walks every source in index.yaml. It decides whether the content actually used it by checking for explicit references or traceable ideas. The process is conservative by default. It is better to miss a borderline source than include one that was not actually used.

The output is a single research.md file. It is fully self-contained, meaning you never need to go back to the memory/ folder again. For this very article, /research_distill should match around 15 to 20 of the 62 sources in the memory folder.

This matters because downstream generation loops re-load the research on each iteration. For example, within the evaluator-optimizer pattern, the system generates, critiques, and revises [8]. Keeping the anchor research small is the difference between an article that stays grounded and one that starts hallucinating.

As I explained in my article on Recursive Language Models (RLMs), when the corpus fits in context with progressive disclosure, fancy retrieval is overkill [9].

What’s Next

For personal-scale research involving hundreds of sources, a well-structured memory/ folder with an index.yaml beats a RAG pipeline on every axis. It gives you full lineage back to source URLs, portability to pass the folder to any agent, and lower costs with no embedding model or vector store.

To further optimize the system, making it more context-efficient, I am considering moving the deduplication and re-ranking fully into Python scripts, adding a local cross-encoder reranker to avoid LLM calls for scoring, and extending the researcher with tag-aware filtering.

But here is what I’m wondering:

What data source in your work makes you most want a private deep research agent? Is it your Obsidian vault, your Readwise library, a code repository, or your team’s shared documents?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

References

1. Govindarajan, V. (2026). OpenClaw Architecture Part 3 - Memory and State Ownership. The Agent Stack. https://theagentstack.substack.com/p/openclaw-architecture-part-3-memory

2. Talebi, S. (2026). Claude Skills Explained in 23 Minutes. YouTube. https://youtube.com/watch?v=vEvytl7wrGM

3. Bowne-Anderson, H. (n.d.). Episode 70: 1,400 Production AI Deployments. Vanishing Gradients Podcast. https://read.readwise.io/read/01kh8p44e70a1273g7ykgx7h5y

4. Huntley, G. (n.d.). Ralph Wiggum as a “software engineer”. ghuntley.com. https://ghuntley.com/ralph/

5. Anthropic. (n.d.). Building More Efficient AI Agents. Anthropic Blog. https://www.anthropic.com/engineering/building-more-efficient-ai-agents

6. Griciūnas, A. (n.d.). Agent Skills: Progressive Disclosure as a System Design Pattern. SwirlAI Newsletter. https://newsletter.swirlai.com/p/agent-skills-progressive-disclosure

7. LlamaIndex. (n.d.). Did Filesystem Tools Kill Vector Search?. LlamaIndex Blog. https://www.llamaindex.ai/blog/did-filesystem-tools-kill-vector-search

8. Anthropic. (2025). Building Effective AI Agents. Anthropic Blog. https://www.anthropic.com/engineering/building-effective-agents

9. Iusztin, P. (n.d.). Your RAG Pipeline Is Overkill (RLMs). Decoding AI Magazine. https://www.decodingai.com/p/your-rag-pipeline-is-overkill-rlms

Images

If not otherwise stated, all images are created by the author.

I am a builder first, not a writer. Most weeks, the writing strangles the building. This is the exact trap most weekly writers fall into.

When the writing eats the week, the next article has nothing real underneath it. So writers fill the gap with generics, surface-level takes, or invented examples that add noise to an already noisy internet.

The default fix most people reach for is to let AI write it. That fails for the opposite reason. When you put zero thought into the process, AI just industrializes the noise.

The whole point of writing is to share something you actually thought through, built, and learned. If AI writes for you, you publish nothing of value. If you write everything by hand, you don’t have enough time to build something worth publishing.

Both ends starve the loop that actually feeds the business: research, build, and teach.

What I built instead is an agentic AI workflow that automates ~90% of the manual writing pipeline while keeping me as the irreplaceable seed at the top. I provide the research direction and the brain dump that reflects my personal experience.

AI handles distribution speed. I handle thought, taste, and direction. By the end of this article, you will see exactly how the system works.

We will cover my deep research agent, writing workflow with its evaluator-optimizer loop, image style-transfer step, and title & SEO generator. And for the most important part, you will learn where the human-in-the-loop is irreplaceable.

My Workflow: What Stays Human, What Gets Automated

Before showing any architecture, I want to walk you through the manual workflow exactly as I used to run it. This is the boring, honest version. This is what every weekly technical writer secretly does, even if they pretend otherwise.

I used to research the topic for hours or days while taking notes. Then, I would write a high-level outline of the piece. Next, I sketched the first high-level diagram that helped me better visualize the narrative of the piece.

I expanded each outline section into bullet points, creating what I call the article guideline. After that, I wrote the article, edited it, and created the rest of the visuals. Finally, I wrote the title and SEO and copy-pasted everything into Substack.

Now, everything gets automated except two things. I still do an in-depth round of research to understand the topic and collect a few high-quality golden source seeds. This is the fun part. These are mostly pulled from my Readwise reading list, which acts as a curated library I built over time while browsing Substack, YouTube, LinkedIn, X, and more. Then, I use this as a high-quality seed for my deep research agent to expand it and fill any potential gaps.

Second, while researching I do a brain dump of everything I consider relevant on the topic. After wrapping up the research, I refactor the brain dump into an outline that follows an engaging narrative. Then, I do a combination of manual and Claude Code work to expand it with bullet points, creating the article guideline.

Together, those two steps are the seed that makes everything downstream mine. Without them, the pipeline produces generic AI mush.

Before the automated pipeline existed, a 3,000-word article like one of my latest pieces, Agentic Harness Engineering, would eat two to three days of my week running this exact nine-step grind by hand. Now, the same piece takes about a day.

Why this works

Writing prose is a translation step. It turns thoughts into words on a page or boxes in a diagram. Translation is exactly the kind of work LLMs excel at, if you already did the thinking.

If you haven’t, no amount of agent orchestration saves you. AI as a writing tool fails when you put zero thought into your process. It becomes a force multiplier when you use it to distribute your thoughts.

Now, let’s look at how the actual system works.

Understanding The System Architecture

The architecture has five big components plus a memory layer. The contract between them is the artifact each one writes to disk, such as the research markdown, the article guideline, the final article, branded image PNGs, and the final HTML bundle.

Here are the five components at a glance:

1. Deep Research agent (we call it Nova): Takes a topic and golden sources, returning a ranked, structured research file.

2. Writing Workflow (Brown): Takes the article guideline and research, returning the full styled article via an evaluator-optimizer loop.

3. Media style transfer: Because the article contains raw Mermaid diagrams, we apply the Decoding AI brand style.

4. Title and SEO generator: Runs an expand-and-narrow loop to produce the title, subtitle, SEO title, and SEO description.

5. HTML exporter: Converts the final markdown into platform-ready HTML for Substack, Medium, X, or LinkedIn to easily copy-paste the piece.

The handoff contract between components is the filesystem. Each stage reads and writes plain files in a working directory. Internal per-component state lives in databases: PostgreSQL for Nova, and an SQLite checkpointer for Brown.

The artifacts make the pipeline debuggable, resumable, and human-in-the-loop friendly across stages. The databases make each stage individually resumable mid-run. For example, if the writing workflow fails after generating the first draft, we can easily resume without having to spend tokens on rerunning from scratch.

Also, because everything is managed through files, I can open any artifact at any key step, inspect it, edit it, and re-run downstream.

We will show you how we used this system to write one of our latest popular pieces: Agentic Harness Engineering.

I’ve also used the same process to research and write professional lessons for other educational projects, such as our latest Agentic AI Engineering course, as the pipeline adapts to any type of educational business.

Each component is explained in depth in the sections below. Two are MCP servers (Nova and Brown) and three are skills (media style transfer, title & SEO, HTML export).

In terms of concrete economics, the whole process runs at roughly ~$0.30 to $1 per image, mostly in Gemini credits, with the rest of the pipeline costing cents. This article, with 9 images, landed closer to $6, while a leaner piece with a single diagram sits around $1.

Build This Exact Stack Yourself (Product)

Reading about a pipeline is one thing. Building one is another. In my Agentic AI Engineering Course, built with Towards AI, I walk you through this exact stack from scratch.

Nova’s deep research loop, Brown’s evaluator-optimizer built on LangGraph, both served via FastMCP, plus the style-transfer skill, evaluation with Opik, and deployment on Docker, GCP, and GitHub Actions.

34 lessons. Three end-to-end portfolio projects. A certificate. And a Discord community with direct access to industry experts and me.

Rated 5/5 by 300+ students. The first 6 lessons are free:

Start here

Walkthrough: The Artifacts of One Article

Before diving into each component, let’s take a look at the input and output artifacts the pipeline produced while generating the Agentic Harness Engineering article. Here are some trimmed versions of each, as they get pretty large.

outline.md: the hand-written seed, Nova’s input (88 lines)

## Outline
1. Introduction - Why Do We Need a Harness?
	1. Personal story: To be researched
	2. Problem + Agitation: ...
	3. Transformation + Solution: ...
	4. Intuitively, Mitchell Hashimoto has the best definition of a harness: "the idea that anytime you find an agent makes a mistake, you take the time to engineer a solution such that the agent never makes that mistake again."
	5. 200 words
2. What the Hell Is A Harness?
    ...
3. How does a Harness Look?
	1. Key components: LLM, tools, planning loop, context engineering, sandbox, memory, orchestration layer, serving layer, interfaces
	2. The agent loop: Powered by planning techniques like ReAct
    ...
4. Planning & Orchestration
    ...
	4. 200 words
5. Key Tools
    ...
6. Sandbox Environment
    ...
7. Memory
    ...
8. Conclusion - The Future of Harness
    ...

# Resources

1. [My AI Adoption Journey](https://mitchellh.com/writing/my-ai-adoption-journey)
2. ...

The seed is deliberately rough. It contains placeholders like “To be researched”, section blocks that will later be restructured, and hand-picked golden sources that anchor Nova’s first round of research. The idea is to dump ideas without thinking too much about structure while you are in your creative mindset.

research.md: Nova’s output (1377 lines)

article_guideline.md: Expanded outline, Brown’s input (201 lines, 8 sections).

## What We Are Planning to Share

...

## Why We Think It's Valuable

...

## Point of View

I write the article, Paul Iusztin. I am part of a bigger team known as Decoding AI....

----

## Article Outline

1. Why Do We Need a Harness?
2. What the Hell Is a Harness?
3. The Anatomy of a Harness
4. How the Agent Decides What to Do Next
5. The Tools That Let Agents Act
6. Where Agents Run
7. Memory Is Just the Filesystem
8. What's Next

## Section 1 - Why Do We Need a Harness?

...

## Section 2 - What the Hell Is a Harness?

...

- **Hook:** Start with the horse analogy. A horse is powerful on its own, but useless for farming without a harness — the straps, reins, and attachments that let you direct its strength toward useful work (inspiration from Jonathan Gimick from Manning). Same with LLMs: the model has the intelligence, but without tools, memory, state, guardrails, and orchestration, you can't put it to work reliably.
- **The clean definition:** LangChain's formulation is the clearest — **Agent = Model + Harness**. The harness is "every piece of code, configuration, and execution logic that isn't the model itself." The model provides intelligence. The harness makes that intelligence useful.
...

[GENERATE_DIAGRAM] Three levels of engineering: prompt, context, and harness engineering.
...

- **Transition:** Now that you know what a harness is, let's look at all its components and how they fit together at a high level — before diving deeper into each one.

- **Section length:** 300 words

## Section 3 - The Anatomy of a Harness

...

## Section 8 - What's Next

...

The article guideline is deliberately as structured and detailed as possible. The idea is to have enough or even more detail about each section to fill in the requested word budget to ensure the LLM doesn’t fill in any gaps with generalities or, worse, with hallucinations.

article.md: Brown’s final prose (~3,000 published words, 8 sections)

See the Agentic Harness Engineering article we posted a few weeks ago on Substack.

Now let’s see how Nova, the deep research agent, turns the outline and its golden sources into a structured research file.

Deep Research: How Nova Builds the Knowledge Base

Nova is an MCP server exposing ten specialized tools, orchestrated by the client, which is often a harness such as Claude Code or Cursor.

Here is how the overall deep research architecture works:

1. Query generation loop: Nova takes the topic and golden sources, runs gap analysis between the outline and the provided sources with Gemini Pro, and generates the next round of research queries based on what is missing. Three rounds hits the cost-versus-coverage sweet spot.

2. Concurrent retrieval: Each round fans out concurrent Perplexity calls that return only metadata and a summary of each new source.

3. Two-stage filtering: We full-scrape only the top five by a four-dimensional rubric evaluating trustworthiness, authority, relevance, and quality. For the rest of the sources we keep only the summary, which is enough for providing examples such as Anthropic is implementing compaction in Claude Code.

Nova ships one purpose-built tool per source family. We scrape web URLs using Firecrawl, while we ingest GitHub repos through gitingest. We ingest YouTube videos using Gemini Pro directly on the URL without a local download.

For example, this is how I used Nova when writing my harness article. I started with a vague topic about what an agent harness is and why it matters. I handed Nova a seed set of golden-source URLs inside the guideline, including the LangChain harness post, the Anthropic long-running agents piece, and Mitchell Hashimoto’s AI adoption journey. Nova extracted these, scraped each one, and wrote the cleaned content into its working memory.

Nova then ran the three-round gap-analysis loop, fanning out concurrent Perplexity queries aimed at topics the seed sources had not covered. Every raw result was appended to the log. Ultimately, each source is filtered using a set of heuristics and LLMs to ensure we keep only high-quality results.

Finally, Nova compiles everything into the collapsible HTML research file.

The client knows how to leverage all of Nova’s MCP tools through a skill that glues together the ingestion, search, and all the other utility tools into the unified deep research algorithm that takes as input the outline.md file and outputs research.md.

Writing Workflow: How Brown Turns an Idea Into an Article

Brown picks up where Nova left off. Brown is a workflow, not an agent, implemented with LangGraph. We chose a workflow over an agent deliberately: prose generation rewards predictability over exploration.

First, we generate all the required Mermaid diagrams for the article using the orchestrator-worker pattern that looks around the article and spins up a specialized Mermaid-diagram agent based on all the user requests found within the article. These are usually flagged within the article guideline explicitly by stating “generate diagram”, “create a diagram”, or [GENERATE_DIAGRAM]. Next, these diagrams are passed downwards through the generation process. We’ll come back to how they get styled in the Branded Images section. The orchestrator-worker pattern can easily be extended to generate other types of media such as images, videos, or audio.

Next, we control Brown’s voice via the system prompt through three large tricks.

The first one is based on defining a set of six profile classes, each targeting a different family of rules. There are four generic profiles, which are static and agnostic to who is using the tool and what they are doing:

1. Structure Profile: How the prose is physically laid out on the page such as sentence, paragraph, list, and subheading shape.

2. Mechanics Profile: The grammatical scaffolding the writing must respect such as active voice, point of view, and punctuation rules.

3. Terminology Profile: What vocabulary is allowed and what filler is banned such as word choice, sentence phrasing, and descriptive language.

4. Tonality Profile: How the article should feel to the reader such as formality level, voice characteristics, and emotional register.

And two customizable:

1. Character Profile: Who is writing. For example, I added here my biography. This should be adapted per user.

2. Article Profile: Special article characteristics such as the structure, referencing, and citations. This can be swapped to a LinkedIn, Reddit, or X profile to adapt the system to different formats.

The second trick is to force the LLM to respect the article guideline and research over anything else, to ensure the user gets what they expect and that Brown adheres only to the research to avoid hallucinations.

The third trick is to add a set of few-shot examples, which beats anything else because showing works better than telling. For the best quality this should be changed when switching article formats and especially when switching content formats.

After we compile our system prompt, we call Gemini at a 0.7 temperature to produce a first draft with more randomness.

After the first generation pass, we start an evaluator-optimizer loop running a Reviewer node with 0.0 temperature against the guideline, research, and profiles to ensure the draft respects all the expected requirements. The Reviewer node returns a list of structured review objects via Pydantic. If issues are found, we run an Editor node at a 0.1 temperature that applies all these fixes.

The evaluator-optimizer loop runs for a fixed iteration count, not until a quality score is good enough. Because writing, like any creative work, is highly subjective, a single quality score becomes noisy and unpredictable. Empirically, running the loop for a fixed number of iterations yields better results and gives us more control over cost and latency.

Because the article might not be polished enough, we expose editing tools through the MCP server so the user can kick off another review-edit iteration on demand.

Now let’s see how we transform raw Mermaid output into branded diagrams.

Generating Branded Images

Brown produces Mermaid source for every diagram in the article. Mermaid is fast and predictable to generate with LLMs but visually generic. In theory you can customize them. But let’s be honest. They are ugly. Thus, the job of this stage is to keep the structure of the Mermaid diagrams Brown produced while applying a styling layer on top of them.

We use a skill that leverages Gemini’s Nano Banana for the style transfer. The skill takes a file as input and detects all the Mermaid diagrams in it. Then, for each diagram, it runs parallel subagents. Each invokes the Gemini script on the raw Mermaid text and outputs a styled PNG.

Here is the prompt engineering behind the styling:

• The branding is referenced both through a written file with color codes, fonts, and general guidelines, plus a representative image.

• 2 positive examples containing both the Mermaid inputs and positive styled outputs.

• 2 negative examples also containing the Mermaid inputs and faulty styled outputs.
  

When using images as few-shot examples, you should be really careful not to go overboard with them, as they add up in tokens quickly. Also, adding the positive and negative examples on top of just random style images and files was the special sauce for us that made everything work, as it clearly shows Nano Banana how to make the mapping between the two.

Now, let’s see how we generate punchy titles and relevant SEO.

Generating Title & SEO

Title and SEO are the most important components. They decide whether the article gets read at all. Doing it by gut on a Friday night is the worst possible workflow.

The pipeline replaces gut with an expand-and-narrow loop. We generate nine versions from many angles, score ruthlessly, and keep only the top four. Then we repeat this process three times.

The generator creates nine candidate title, subtitle, SEO title, and SEO description packages per round, each from a different angle like personal transformation, curiosity, making bold claims, showing proof of work, and more. The idea is to have a lot of diversity during the expansion round.

The validator scores every candidate on six rubric-anchored dimensions: title, subtitle, SEO title, and SEO description quality, article alignment, and cohesion across the four pieces. It uses hybrid scoring, combining an LLM-judge for the qualitative rubrics and heuristic penalties for the hard constraints such as character count. For example, shorter titles score higher.

Then, based on the scores generated by the validator, we pick the top four winners and use them as seeds for the next round of generation.

The key here is to make the validator a subagent that doesn’t share the same context window as the generator to avoid any type of bias. Fresh eyes prevent self-confirmation bias. This is the same principle Brown uses for its evaluator-optimizer split and Nova uses for its filter step.

So why pick the top four, and not three? When scheduling on Substack, we pick the top four for A/B testing rather than committing to the single highest-scored one. The validator is good but not omniscient, so we let real readers settle close calls.

Exporting to HTML

The last step is to compile the Markdown article into HTML so we can easily copy-paste everything into Substack. Boring, but necessary.

For this step, we created a skill that wraps ’s nb2wb CLI tool that does all the heavy lifting. The tool supports most popular formats such as Substack, Medium, X, and LinkedIn.

Initially, it was built to map Jupyter Notebooks to these formats, but it works amazingly for Markdown files too.

What Stays Irreplaceable

It is not 100% automated. I still follow the original research direction. I still write the outline brain dump. I still validate every artifact. I still write the code that runs the pipeline.

The 90% automation is real, but the 10% is the part that matters most. It’s the part that makes this article stand out as human: the seed, the taste, and the validation are irreplaceable.

What’s Next

You might wonder how well this works. Well... You just read an article created by this exact workflow. In other words, this is an article that talks about itself. It’s not yet perfect, but it will get there.

End-to-end, it took about a day of my time. Without the pipeline, this same article would have taken three days of mostly translation work.

💡 Want to build this exact stack yourself? Nova and Brown built with FastMCP & LangGraph, the style-transfer skill, human-in-the-loop orchestration, evaluation with Opik, and deployment on Docker, GCP, and GitHub Actions. Every line walked through with me and the Towards AI team. That’s exactly what we teach in our Agentic AI Engineering Course.

Otherwise, here is what I’m wondering:

Which step of your own writing workflow do you think is the most dangerous to automate, and which one have you been avoiding automating because you weren’t sure how?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

Further Reading

1. LangChain. The Anatomy of an Agent Harness

2. Anthropic. Effective Harnesses for Long-Running Agents

3. Mitchell Hashimoto. My AI Adoption Journey

4. The Agent Stack. OpenClaw Architecture Part 1

5. cefboud. How Coding Agents Actually Work: Inside OpenCode

Images

If not otherwise stated, all images are created by the author.

In most AI projects, such as the financial assistant I am working on, there is a constant battle between Retrieval-Augmented Generation (RAG) and Context-Augmented Generation (CAG). Should you implement a heavy RAG architecture up front that might not even work, or does CAG get the job done? For example, in our financial assistant system, we ultimately decided to use RAG only when we really HAVE to, because it introduces zigzag retrieval patterns that require dozens of queries per operation, increasing latency.

Also, while building Brown, my writing agent, I hit another wall. Brown needs to ingest massive amounts of research to anchor its writing process. At 180,000 input tokens, the Gemini API became entirely unreliable.

I faced constant timeouts, disconnections, and infrastructure breakdowns. Huge context windows suffer from API reliability and infrastructure stability issues, as well as performance degradation. But the thing is, I didn’t want to overcomplicate my solution with a RAG layer, so I started looking around for other solutions.

Most engineers face this painful tradeoff when working with large documents. You can stuff everything into the context window, but performance degrades quickly. This causes context rot, which happens when attention degrades over long contexts and earlier information loses its influence [1], [2].

Alternatively, you can build a RAG pipeline. But that requires maintaining vector databases, chunking strategies, and retrieval evaluation infrastructure.

Even the tools we use daily, like Claude Code or Cursor, rely on summarization-based context compression that loses critical information. I just wanted to dump my research into one file and get good answers without the infrastructure breaking. Recursive Language Models (RLMs) solve this exact problem [3].

RLMs use an inference-time pattern that treats your input as an external environment the model interacts with programmatically. You do not need chunking infrastructure or embedding pipelines. The model writes code to explore, filter, and recursively process your data on demand.

This approach scales the effective input and output lengths of LLMs. Researchers tested RLMs up to 10 million tokens across GPT-5 and Qwen3-Coder, showing they easily outperform base models [3]. Base model performance degrades as a function of input length and task complexity, while RLM performance scales with less degradation.

RLMs are also a model-agnostic inference strategy, meaning they work with any model you choose.

However, this architecture has honest downsides you must consider. The inference cost has high variance due to differences in trajectory lengths. The system suffers from code fragility, meaning that if the model writes buggy code, the entire reasoning chain fails.

Errors in sub-calls can compound through the recursive tree, propagating hallucinations. Sequential sub-calls also create latency bottlenecks. This makes RLMs best suited for deep thinking applications rather than real-time chat.

To understand how we bypass these infrastructure limits, we need to examine the specific programming trick that keeps the model’s memory clean.

Here is what you will learn about this pattern:

• The mechanism that keeps massive documents outside the context window.

• The orchestration loop that drives programmatic data exploration.

• The specific use cases where this pattern outperforms retrieval systems.

• A practical method to approximate this behavior using Claude Code.

If You Want To Go Deeper Into Production AI (Product)

Patterns like RLMs show that the real challenge isn’t the model, but the infrastructure and systems around it, called the harness. If you want to master that harness, check out my Agentic AI Engineering course, built with Towards AI.

34 lessons. Three end-to-end portfolio projects. A certificate. And a Discord community with direct access to industry experts and me.

Rated 5/5 by 300+ students. The first 6 lessons are free:

Start here

The REPL Trick That Keeps Your Context Window Clean

RLMs introduce a simple core idea. Do not feed the document into the model’s context window. Instead, load it as a variable in a persistent programming environment and let the model write code to interact with it [4].

The model never sees your 10-million-token document directly. In a traditional agent, the prompt goes into the model, completely blowing up your context window. In an RLM, the context stays outside as an external variable, and the model receives only a symbolic handle to it.

The system initializes a Read-Eval-Print Loop (REPL), which is a persistent interactive programming environment where variables and state persist across iterations [3].

The root model receives only metadata, such as the total character count and data structure. It also receives instructions on how to access the REPL. The model then writes code to peek into, filter with regex, chunk, or summarize the data.

When the model identifies a sub-task, it uses a specific primitive such as llm_query(prompt, chunk) to spawn a fresh, isolated worker sub-model [3]. The system pauses, executes this sub-call, and returns the result to the root model’s REPL.

Variables persist across these REPL turns. The model aggregates findings into a buffer, building the response progressively across iterations. Once confident, it calls FINAL(answer) to stop the recursive loop and return the response [5].

RLMs essentially perform context engineering on autopilot. Traditional context engineering requires you to carefully curate what goes into the context window through retrieval and compression [1]. RLMs automate this by letting the model itself decide what to extract, filter, and process.

Costs and performance stay intact because the model filters the input context without explicitly seeing it. By writing Python scripts, the model processes only the relevant portions through sub-calls. Only constant-size metadata about execution results is appended to the root model’s history, keeping its context window small and clean.

Understanding this mechanical loop allows us to map the pattern directly to production harness engineering.

Turn Any Agent Into a Plan-Execute-Validate Machine

RLMs are an inference-time orchestration pattern that maps directly to production harness engineering. If you have built agent systems, you already know the components: a planning loop, tool execution and validation [7]. RLMs formalize this into a programmable, recursive architecture.

A robust RLM harness uses a multi-tiered architecture. The root controller is a frontier model that acts as the project manager. It plans the reasoning process, writes code, and coordinates execution, but never directly interacts with tools or the full document [8].

Worker sub-models are cheaper, faster models spawned via an operation such as llm_query() to handle specific, localized sub-tasks. This reduces overall costs while maintaining high quality. The aggregation layer is the REPL environment that combines recursive step results into a final structured response via persistent variables.

This setup naturally follows the plan-execute-validate mapping. In the plan phase, the root controller reviews the query, creates a reasoning plan, and decides how to decompose the problem. It might plan to regex-filter a codebase, chunk a document, or batch sub-calls for parallel analysis.

In the execute phase, the model translates the plan into code. It writes Python scripts, issues llm_query() calls, and spawns worker sub-models for parallel execution in isolated REPL environments. External tools, like web search, are provided ONLY to worker sub-models, keeping the root model’s context perfectly clean.

After execution, the system enters the validation phase, where results feed back as observations. The root model assesses accuracy, launches verification sub-calls, and handles errors by dynamically adjusting its plan. If the Python code fails, the error traceback is yielded back to the model as an event.

This allows the model to adapt and fix its code on the next turn. The cycle repeats until the model calls FINAL(answer).

Deploying this in the real world requires strict production guardrails. You must configure maxIterations to cap the number of REPL turns, typically between 10 and 50. You need maxDepth to limit the recursive stack depth, where a depth of 1 is usually sufficient.

You also need maxStdoutLength to truncate REPL output returned to the model to prevent context overflow. Finally, permission gating is required to provide sandboxed execution with explicit approval for sensitive operations.

Neither Claude Code nor OpenAI Codex uses true RLM patterns. They rely on summarization-based context compression, file-system state tracking and progressive disclosure techniques [9]. This creates a succession of agents connected by prompts and file state, rather than maintaining a persistent REPL environment with programmatic sub-calls.

With this architecture in place, we can identify the specific real-world scenarios where this pattern outperforms traditional data processing.

Four Scenarios Where RLMs Beat Traditional Approaches

RLMs are best suited for deep thinking applications that require accuracy, multi-step reasoning, and reliability over massive contexts. They are not suited for real-time, low-latency chat applications.

The first scenario is parsing large files without building retrieval infrastructure. Instead of building a hybrid index with vector and graph search, you keep everything in one file or directory and use an RLM agent to extract information on demand.

We can view the relationship between RAG and RLMs as a spectrum. For simple cases, RLMs replace RAG entirely, removing the need for chunking and embeddings. For advanced scenarios, RLMs complement retrieval beautifully.

You use semantic search to find your first pool of candidates, write the results to disk as cached short-term memory, and use an RLM to query that refined dataset on demand.

The retrieval narrows the haystack, and the RLM reasons deeply over what is left. I use this exact workflow for my research, dumping everything into a massive text file and using an RLM to extract relevant information.

The second scenario is complex software engineering and codebase comprehension. RLMs ingest massive codebases containing millions of tokens to answer questions about architecture, map dependencies, and perform reviews.

The RLM paper tested this on LongBench-v2 CodeQA using Qwen3-Coder with a Python REPL. The model writes code to break down the codebase, launches sub-queries to smaller language models, and aggregates findings [3].

The third scenario is enterprise legal and financial analysis. RLMs provide consistent interpretation across thousands of contracts, case files, and policies that would overwhelm a standard context window. They also excel at financial audits and due diligence by tracing, validating, and reasoning through massive financial datasets.

The fourth scenario is deep research and information synthesis. RLMs synthesize research across thousands of files by programmatically filtering, chunking, and summarizing. They enable knowledge graph exploration and multi-hop reasoning over large document dumps.

At scale, RLMs become both more accurate and cheaper than standard long-context approaches. They avoid paying for n-squared attention over massive contexts by having the model process only relevant slices via sub-calls. In all these scenarios, the RLM pattern succeeds because it treats the LLM as a project manager that decides what to look at and delegates sub-tasks to workers.

Knowing these optimal use cases helps us approximate the pattern using tools you likely already have installed.

Build a Naive RLM SKILL in Claude Code

Claude Code does not natively use the RLM pattern. It relies on summarization-based context compression, file-system state tracking, and progressive disclosure. However, you can approximate RLM behavior using Claude Code’s existing harness features to build a naive RLM SKILL.

First, you set up the environment by having the SKILL load the target file or directory as a reference. Instead of feeding it into the context window, it writes the file path and metadata to a prompt for the root agent.

Second, the root Claude Code agent receives only this metadata and a set of instructions for how to interact with it. It uses its Explore subagent type
to examine the data structure, identify relevant sections, and plan its approach.

Third, the SKILL uses Claude Code’s Agent tool to spawn subagents. Each subagent receives a focused prompt to read specific lines and extract mentions, returning a condensed summary of a few thousand tokens. This mirrors the RLM pattern of spawning isolated sub-calls that process slices of the input.

Finally, the root agent collects these subagent results. It aggregates them into a coherent answer and decides whether more exploration is needed or whether to finalize the output.

Here is what this naive RLM SKILL looks like as a SKILL.md file:

---
name: rlm-research-analyzer
description: "Analyze large research files by treating
  them as an external environment. Instead of stuffing
  content into context, the model explores, decomposes,
  and recursively processes the data through subagents."
---

# Analyze Large Research Files Using the RLM Pattern

## Step 1 — Initialize the environment

Accept the target file path as an argument. Do NOT read
the file into context. Instead, run a Bash command to
collect metadata:

wc -l <file_path>   # total lines
wc -c <file_path>   # total bytes
head -5 <file_path>  # short prefix

Write the metadata and file path to a temporary prompt
file at <working_dir>/rlm_prompt.md. The root agent
receives ONLY this metadata, never the full content.

## Step 2 — Plan the exploration

Read rlm_prompt.md. Based on the metadata and prefix,
decide how to decompose the file. Use an Explore
subagent to scan the file structure:

- Identify section boundaries, headings, or delimiters
- Estimate which regions are relevant to the query
- Produce a ranked list of target ranges to process

## Step 3 — Delegate to worker subagents

For each target range, spawn an Agent subagent with a
focused prompt:

"Read lines {start}-{end} of {file_path}. Extract all
findings related to {query}. Return a summary under
2000 tokens."

Launch multiple subagents in parallel when ranges are
independent. Write each subagent's output to
<working_dir>/slice_{n}.md.

## Step 4 — Aggregate and finalize

Read all slice files. Synthesize the findings into a
single coherent answer. If gaps remain, return to
Step 3 with new target ranges. Otherwise, write the
final output to <working_dir>/answer.md and present
it to the user.

Notice how the four steps map directly to RLM primitives. Step 1 mirrors REPL initialization, where the data becomes an external variable rather than context input. Step 3 replaces the theoretical llm_query() operation with Claude Code’s Agent tool. Step 4 mirrors the FINAL() call that terminates the recursive loop.

This naive approximation lacks several critical features. It has no true REPL persistence, as Claude Code subagents do not share a persistent variable space. The filesystem serves as a proxy for REPL state, but it is slower and less elegant.

It also lacks sandboxing, as Claude Code runs directly in your environment. Then you miss out on configurable guardrails like max_iterations and max_output_chars, requiring manual limits instead. You get the idea.

Still, I’ve been using a similar technique in all my current projects: instead of stuffing the research into a file, I dump everything into a dir and link everything together in an index.yaml file that contains URIs to all the files, plus metadata such as the title and a 1-2 sentence summary of each source. Like this, through the index.yaml file, Claude Code can efficiently navigate the whole research dump token through progressive disclosure.

My structure looks something like this:

research/
├── index.yaml
├── file_1.md
├── file_2.md
├── ...
└── file_N.md

Also, the only out-of-the-box implementation I found is within the DSPy framework.

The naive SKILL is a useful thought exercise and a practical first step. For production use, you should reference the DSPy framework’s dspy.RLM module.

What’s Next

RLMs represent a fundamental shift in how we process large inputs. We are moving from asking how to fit data in the context window to asking how we let the model interact with it programmatically. This is a great thought exercise on integrating specialized inference-time functionality into your harness.

As models get better at writing code and REPL environments become more sophisticated, the boundary between the model and its infrastructure will blur. The model does not just use tools, it writes the tools on the fly to solve the specific problem in front of it.

Your next practical step is to experiment with our SKILL or with the DSPy framework’s dspy.RLM module on a real problem. Point it at a large codebase you need to understand or a research corpus you need to synthesize. Start with something you have been using RAG or context stuffing on, and see whether the RLM approach is more effective.

But here is what I’m wondering:

How have you been passing large files, such as deep research results or books, to your agents so far? RAG, CAG or other creative techniques?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

Whenever you’re ready, here is how I can help you

If you want to go from zero to shipping production-grade AI agents, check out my Agentic AI Engineering course, built with Towards AI.

34 lessons. Three end-to-end portfolio projects. A certificate. And a Discord community with direct access to industry experts and me.

Rated 5/5 by 300+ students. The first 6 lessons are free:

Start here

Not ready to commit? Start with our free Agentic AI Engineering Guide, a 6-day email course on the mistakes that silently break AI agents in production.

References

1. (n.d.). Effective Context Engineering for AI Agents. Anthropic. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

2. (n.d.). MIT’s new ‘recursive’ framework lets LLMs process 10 million tokens without context rot. VentureBeat. https://venturebeat.com/orchestration/mits-new-recursive-framework-lets-llms-process-10-million-tokens-without-context-rot/

3. Zhang, A. L., Kraska, T., & Khattab, O. (2025). Recursive Language Models. arXiv. https://arxiv.org/abs/2512.24601

4. (n.d.). Recursive Language Models: the paradigm of 2026. Prime Intellect. https://www.primeintellect.ai/blog/rlm

5. (n.d.). Why Recursive Language Models (RLMs) Beat Long-Context LLMs. Dextra Labs. https://dextralabs.com/blog/recursive-language-models-rlm/

6. Mansurova, M. (2026, March 30). Going Beyond the Context Window: Recursive Language Models in Action. Towards Data Science. https://towardsdatascience.com/going-beyond-the-context-window-recursive-language-models-in-action/

7. (2026, March 21). The Anatomy of an Agent Harness. LangChain Blog. https://blog.langchain.com/the-anatomy-of-an-agent-harness/

8. (2025, December 24). Building Effective AI Agents. Anthropic. https://www.anthropic.com/engineering/building-effective-agents

9. (2026, March 25). Effective Harnesses for Long-Running Agents. Anthropic. https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents

Images

If not otherwise stated, all images are created by the author.

When we stripped everything back to plain Python, simple API calls, and a custom ReAct engine, things finally worked. What we accidentally built was a harness featuring specialized financial tools, domain-specific guardrails, and purpose-built context engineering.

We did not know the term yet, but the lesson was clear. The model was never the problem. The system and infrastructure around it were.

Most engineering teams obsess over which model to use. They debate GPT-4o versus Claude Opus versus Gemini. They chase LLM benchmark scores and swap models, hoping for better results.

But the model is only half the equation. The system and infrastructure around it determine whether your agent actually works in production.

TerminalBench 2.0 proved this. Changing only the harness moved the DeepAgent from LangChain from outside the top 30 to the top 5 [1].

This is what usually happens. You have a powerful model. You gave it tools and a prompt. It works in demos.

But shipping it to production means solving problems the model cannot solve alone. You must bridge context windows, recover from failures, serve multiple interfaces, and manage state across sessions.

The solution is harness engineering. This is the discipline of building the infrastructure around the model so it can do useful work reliably. As Mitchell Hashimoto noted, harness engineering is the practice of engineering a solution every time an agent makes a mistake, ensuring it never makes that specific mistake again [2].

By the end of this article, you will learn:

• What an agent harness actually is.

• The core components powering production AI systems.

• How the planning loop dictates agent actions.

• The design principles behind an effective toolset.

• How to manage memory using the filesystem.

Before we look at all its components and how they fit together, we must first define what a harness actually is.

Your Path to Agentic AI Engineering for Production (Product)

Most engineers know the theory behind agents, context engineering, and RAG. What they lack is the confidence to architect, evaluate, and deploy these systems in production. The Agentic AI Engineering course, built in partnership with Towards AI, closes that gap across 34 lessons (articles, videos, and a lot of code).

By the end, you will have gone from “I built a demo” to “I shipped a production-grade multi-agent system with evals, observability, and CI/CD.” Three portfolio projects, a certificate to back them up in interviews, and a Discord community with direct access to industry experts.

Rated 5/5 ⭐️ by 300+ early students saying “Every AI Engineer needs a course like this” and that is “An excellent bridge from experimental LLM projects to real-world AI engineering.”

Start learning today. The first 6 lessons are free:

Enroll here

So... What the Heck Is a Harness?

While talking with Jonathan Gennick from Manning, he said that the first time he heard about the term “harness” was in the context of horses. Let me explain. A horse is powerful on its own, but useless for farming without a harness. The straps and reins let you direct its strength toward useful work. The same applies to LLMs.

The model has intelligence. But without tools, memory, state, guardrails, and orchestration, you cannot put it to work reliably.

LangChain offers the clearest definition. An agent equals a model plus a harness. The harness is every piece of code, configuration, and execution logic that is not the model itself [1].

A basic agent, as we know it so far, is just a model, a prompt, tools, and a planning loop. A harness extends this by adding memory systems, guardrails, advanced orchestration, context engineering, and multi-agent coordination.

Usually, it also includes a serving layer that connects the agent to various user interfaces, such as terminal apps, web dashboards, IDE plugins, and messaging apps like Telegram.

Ultimately, a harness is a term for building real software applications using LLMs or other models as the operating system. Applications like Claude Code, OpenCode, OpenClaw, and Codex are all harnesses. You could swap the model inside them, but the real engineering value lives in the harness itself.

This introduces three distinct levels of engineering. Prompt engineering crafts the instructions. Context engineering dictates what goes into the context window and when.

Harness engineering is the full application and infrastructure. It controls when context loads, which tools are available, which actions are allowed, and how failures are handled. Each level encompasses the previous one [3].

Now that you understand what a harness is, the next step is to explore the internal architecture and see how these pieces connect.

The Anatomy of a Harness

A complete harness consists of the LLM, tools, a planning loop, context engineering, a sandbox, memory, an orchestration layer, and a serving layer. In other words, everything that has been hovering within the AI space is finally falling into one beautiful system.

One of the most distinctive features of modern harnesses is the multi-surface architecture. OpenClaw serves the same agent across a command-line interface (known as TUI), a web UI, desktop apps, Slack and Telegram/WhatsApp through a centralized Gateway using a typed WebSocket protocol.

Codex evolved from a simple terminal tool to an App Server using JSON-RPC over standard input and output. OpenCode uses a Bun JS HTTP server where any client connects via HTTP, utilizing an Event Bus to broadcast results in real-time [4], [5], [6].

This architecture introduces challenges. Multiple messages arrive in parallel from different clients. Users ask questions while the model is still processing.

To solve this, systems use priority queues and message buses. OpenClaw uses a lane-aware FIFO queue to ensure only one active run per session while allowing parallelism across different sessions.

At the core of all this infrastructure, the filesystem is king. As the most foundational harness primitive, it enables durable storage, workspace management, multi-agent collaboration, and versioning.

You heard me right, there is no fancy vector database in place. With AI, we are going back to basics, and nothing is purer than the filesystem itself.

Every production harness uses the filesystem as its primary state mechanism [1].

You might wonder if this is just traditional orchestration like Airflow. It is different in three key ways. The agent loop is non-deterministic, context management is a first-class concern, and the programmer inside the loop is the LLM itself. It is common to add durability to the harness using tools such as Prefect, Temporal or DBOS that natively support dynamic pipelines rather than predefined, rigid DAGs.

Let us zoom in on the first and most fundamental component: the planning loop.

How the Agent Decides What to Do Next

The most common pattern for the planning loop is ReAct, which stands for Reasoning and Acting. The model receives the current state, reasons about what to do next, takes an action via a tool call, and observes the result. This cycle repeats continuously until a strict stopping condition is met [5].

Consider a concrete example. A user asks the agent to fix a failing test. First, the model reads the test output, reasons that the import path is wrong, and edits the file through a tool.

Second, it re-runs the tests, sees a new type mismatch error, and fixes it. Third, it runs the tests again.

They pass, the model reasons the job is done, and it stops. The harness orchestrates this loop, while the model reasons and picks actions.

When tasks are too complex for a single agent, harnesses use orchestrator-worker patterns. The orchestrator decomposes a task, delegates subtasks to specialized workers, and aggregates the results.

In OpenCode, a dedicated task tool spawns subagents. Each subagent gets its own session, context window, and restricted tool set [7].

For tasks that span multiple context windows, Claude Code implements Ralph Loops. This harness mechanism intercepts the model’s attempt to exit via a hook. It reinjects the original prompt in a clean context window, forcing the agent to continue against a completion goal using the state persisted on the filesystem [1].

While automating my business with agents, I learned a hard lesson about orchestration. I initially built five specialized agents, each handling one step.

I eventually found that a single agent with memory and smart context engineering outperformed the whole swarm. Always start with one well-harnessed agent before reaching for multi-agent complexity.

Here is a deep dive into planning:

While the planning loop decides the next step, the agent still needs a way to interact with its environment.

The Tools That Let Agents Act

This interaction happens through a specific toolkit designed for autonomous execution.

First, Bash is a general-purpose tool. The agent can run any shell command to execute tests, linters, or builds. This gives the model code execution capabilities so it can design its own tools on the fly rather than being constrained by fixed options.

For example, the agent runs Python code and executes it through python -c "...", generates a script and runs it through python main.py or runs your code as python -m my_module.main.

Second, specialized filesystem tools handle common operations like reading, writing, editing, and searching. Doing file operations via Bash is slow and error-prone.

Specialized tools include safety checks. For instance, a read tool enforces absolute paths and line limits, while an edit tool validates the uniqueness of replacement strings.

Third, state management tools track session-scoped tasks. These give the agent working memory within a single session. For example, OpenCode has ToDoAdd and ToDoRead tools that add/read tasks from a queue to keep track of the plan it has to execute.

Finally, orchestration tools launch subagents with their own isolated prompts and context windows, such as OpenCode’s task tool or Claude Code’s agent tool.

Feedback loops are the most important principle around tooling. Boris Cherny, the creator of Claude Code, noted that giving the model a way to verify its work improves quality by two to three times. For example, OpenCode integrates the Language Server Protocol (LSP) to get real-time code definitions and diagnostics.

Undefined variables and type errors are fed back to the LLM for immediate correction. These tools do not act on the world. They feed vital information back to the planning loop.

Harnesses also enforce tool access control. In OpenCode, the planning agent cannot call edit tools. This prevents exploratory agents from accidentally modifying your code [5].

Here is a deep dive into tool calling:

Once the agent has its tools, it needs a secure place to use them. In production, this requires strict isolation.

Where Agents Run

Agents execute code, and that code can fail, crash, or delete all your files. I know I want my precious notes protected. Sandboxes isolate agent execution so failures do not affect the host system or other agents. The cherry on top is that they also enable horizontal scaling across parallel environments.

There is a strict tradeoff between security and capability. Not every harness uses the same approach. Codex uses a hard sandbox.

Each task runs in an isolated cloud container preloaded with the repository. This provides maximum safety, but the agent cannot access the host filesystem [6].

Conversely, OpenClaw uses a soft sandbox. The workspace is the default working directory. This grants maximum capability but introduces more risk.

OpenClaw deliberately avoids hard sandboxing to preserve full filesystem access. Most production harnesses sit somewhere between these extremes, depending on the trust model.

When you submit a task to Codex, the harness spins up a fresh cloud container. The agent works inside this container to read files, run tests, and install packages.

It cannot touch your local machine. When the job finishes, the results are extracted, and the container is destroyed.

Along with security, a major benefit of cloud sandbox environments is that they give the agent access to powerful computing resources. For example, if you want to train a model using a GPU, you can ask the agent to implement and run a training pipeline hosted in a sandbox powered by a GPU.

This is similar to manually SSHing to different VMs and running the code manually there. Based on the same principles, you can easily spin up multiple cloud sandboxes and run your agents in parallel.

On the other side of the spectrum, you can also run sandbox environments locally through Docker containers or isolated processes, similar to what Cursor does. Super useful when you want to try something out and give the agent full permissions to avoid having to supervise it.

While sandboxes provide a safe space for execution, they are ephemeral by design.

Memory Is Just the Filesystem

To survive across sessions and context windows, every harness manages state across three distinct memory layers. The first layer is the filesystem. This is the long-term memory.

It is durable and persistent, surviving across sessions. This is where progress files, git history, and session transcripts live.

The second layer is the RAM. This is the short-term memory, also known as the working memory. It holds the conversation history and tool results during an active session. It is fast but volatile.

The third layer is the context window. This is what the model actually sees. It is the strictest constraint, as everything the model knows about the current task must fit here.

The harness orchestrates the dynamics between these layers. On the read path, the harness selectively loads relevant state from the disk into the RAM.

It then assembles the context window using context engineering techniques such as compaction, progressive disclosure, and just-in-time retrieval. On the write path, the harness persists important state back to the disk after processing.

OpenClaw enforces a strict invariant that memory is always flushed to disk before being discarded from context. Rehydration is treated as a tool-shaped action, where the agent searches and then retrieves specific data, rather than dumping everything into the context window [8].

Context engineering makes this possible. When token counts exceed ninety percent of the limit, OpenCode automatically summarizes the conversation. Codex assembles prompts from multiple sources and exploits prompt caching.

Anthropic recommends using structured note-taking files and sub-agent architectures to isolate context [5], [6], [9].

In Anthropic’s long-running agent pattern, an initializer agent creates a script, a progress file, and a feature list. The coding agent reads the git logs and progress files at the start of each session and updates the progress file as it progresses.

The beauty? There is no database or vector store. It is just the filesystem [10].

Here is a deep dive into memory:

Now that you have seen all the pieces, from planning and tools to sandboxes and memory, the question is what this means for how you build software.

What’s Next

We are witnessing a new way of building software. Instead of software engineers building traditional frontend and backend applications, the next generation of production software will be harnesses. Harness engineering is merging software engineering with AI, moving it one level up [3].

Popular tools like Claude Code are just the beginning. In the long run, no company will want to depend entirely on proprietary harnesses. Even open-source solutions like OpenCode will not cover every specific use case.

Companies will inevitably build their own. As we experienced at ZTRON, custom systems and infrastructure are what finally make an agent work in production.

However, we must be honest about current limitations. Memory still breaks across long sessions. Validation loops still miss edge cases. Furthermore, orchestrating hundreds of parallel agents on shared codebases remains an open research problem.

Harness engineering is real engineering. Your harness becomes its own product with its own bugs, its own drift, and its own maintenance burden.

What’s your opinion? Do you agree, disagree, or is there something I missed?

Leave a comment

Enjoyed the article? The most sincere compliment is to share our work.

Share

Go Deeper

Go from zero to production-grade AI agents with the Agentic AI Engineering self-paced course. Built in partnership with Towards AI.

Across 34 lessons (articles, videos, and a lot of code), you’ll design, build, evaluate, and deploy production-grade AI agents end to end. By the final lesson, you’ll have built a multi-agent system and a capstone project where you apply everything you’ve learned on your own.

Three portfolio projects and a certificate to showcase in interviews. Plus a Discord community where you have direct access to other industry experts and me.

Rated 4.9/5 ⭐️ by 300+ students saying “Every AI Engineer needs a course like this.”

Learn more

Not ready to commit? We also prepared a free 6-day email course to reveal the 6 critical mistakes that silently destroy agentic systems. Get the free email course.

References

1. LangChain. (2026, March 21). The Anatomy of an Agent Harness. LangChain Blog. https://blog.langchain.com/the-anatomy-of-an-agent-harness/

2. Hashimoto, M. (2026, March 25). My AI Adoption Journey. Mitchell Hashimoto. https://mitchellh.com/writing/my-ai-adoption-journey

3. Bouchard, L. (2026, March 25). What Harness Engineering Actually Means. What’s AI by Louis-François Bouchard. https://youtube.com/watch?v=zYerCzIexCg

4. Govindarajan, V. (2026, March 21). OpenClaw Architecture Part 1 - The Agent Stack. The Agent Stack. https://theagentstack.substack.com/p/openclaw-architecture-part-1-control

5. Abboud, M. (2026, March 17). How Coding Agents Actually Work: Inside OpenCode. Moncef Abboud. https://cefboud.com/posts/coding-agents-internals-opencode-deepdive/

6. ByteByteGo. (2026, March 26). How OpenAI Codex Works. ByteByteGo. https://blog.bytebytego.com/p/how-openai-codex-works

7. Anthropic. (2025, December 24). Building Effective AI Agents. Anthropic. https://www.anthropic.com/research/building-effective-agents

8. Govindarajan, V. (2026, March 24). OpenClaw Architecture Part 3: Memory and State Ownership. The Agent Stack. https://theagentstack.substack.com/p/openclaw-architecture-part-3-memory

9. Anthropic. (2025, October 22). Effective Context Engineering for AI Agents. Anthropic. https://www.anthropic.com/engineering/effective-context-engineering

10. Anthropic. (2026, March 25). Effective Harnesses for Long-Running Agents. Anthropic. https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents

Images

If not otherwise stated, all images are created by the author.

It is 2026. People across the industry still mix up words like workflows, agents, tools, and multi-agent systems. Beyond terminology, this confusion has led to massively overengineered solutions.

Teams jump to multi-agent architectures because it sounds impressive and helps raise money. In reality, a simple workflow would have been faster to build, cheaper to run, and easier to debug. The result is bloated systems, wasted tokens, and debugging nightmares.

Our goal is to provide a clear mental model of what architecture to choose for your AI project: workflows vs. single agents vs. multi-agent systems.

from Towards AI has been working on this exact problem with his clients and distilled his decision framework into two YouTube videos: Stop Overengineering: Workflows vs AI Agents Explained and From Workflows to Multi-Agent Systems: How to Choose. He allowed me to take that framework and turn it into this article. Kudos to Louis-François!

This decision framework is a spectrum from simple to complex that tells you exactly what to build based on your actual constraints. The goal is to stay as far left on the complexity spectrum as possible while still solving your problem.

Here is what you will learn:

• The fundamental difference between an agent and a workflow.

• How to use the complexity spectrum to make architecture decisions.

• When to rely on simple workflows for predictable tasks.

• Why a single agent with tools is often enough for dynamic problems.

• The exact breaking points that justify moving to a multi-agent system.

To apply this spectrum effectively, you must first define the terms. Here are the core misconceptions that lead to bad architecture decisions.

Before we continue, a quick word from the Decoding AI team. ↓

Go Deeper: Your Path to Agentic AI for Production

Most engineers know the theory behind agents, context engineering, and RAG. What they lack is the confidence to architect, evaluate, and deploy these systems in production. The Agentic AI Engineering course, built in partnership with Towards AI, closes that gap across 34 lessons (articles, videos, and a lot of code).

By the end, you will have gone from “I built a demo” to “I shipped a production-grade multi-agent system with evals, observability, and CI/CD.” Three portfolio projects, a certificate to back them up in interviews, and a Discord community with direct access to industry experts like and me.

Rated 4.9/5 ⭐️ by 300+ early students saying ”Every AI Engineer needs a course like this” and that is ”an excellent bridge from experimental LLM projects to real-world AI engineering.”

Start learning today

↓ Now, back to the article.

Clarifying the Confusion: Not Everything Is an Agent

The first major misconception is that every LLM application is an agent. The key difference is autonomy. In a workflow, you control the flow.

You decide the steps and their order. In an agent, the model controls the flow. It decides what to do next based on the goal you give it.

If you can write down the exact sequence of steps in advance, you are building a workflow. You are not building an agent.

The second misconception is that tools are agents. A tool is a capability. It can be a calculator, a database query, a web browser, a validator, or an API call.

It can even be another LLM. An agent is the decision maker who chooses which tools to use and when.

If someone tells you they built a multi-agent system, but it is actually one model calling ten different APIs, that is not multi-agent. That is a single agent with ten tools.

This distinction matters. It defines how you architect, debug, and scale your system. It drives your core architecture choice between a workflow, a single agent with tools, or multiple agents.

The Complexity Spectrum: A Mental Model for Architecture Decisions

To make this architecture choice easier, we use a complexity spectrum. It is a slider going from the most control to the most autonomy. Your goal is to stay as far left as possible while still solving the problem.

Level 1 represents workflows. Here, you chain multiple LLM calls together in a predefined sequence. You control every step.

Level 2 represents a single agent with tools. The model makes decisions about what to do next. You have one decision maker and multiple capabilities.

Level 3 represents multi-agent systems. Here, you have multiple decision makers who need to coordinate with each other.

The core principle is straightforward. Move right on this spectrum only when you absolutely have to. Each step to the right increases costs, latency, and debugging complexity.

More LLM calls mean more tokens, more traces to follow, and more places where things can go wrong.

In practice, start simple and escalate only where things break. Write a prompt first. Test it.

Implement it with minimal complexity. Measure the results. Add what is missing.

If the model lacks information, add retrieval. If it needs calculations, add a tool. Only when you genuinely need autonomous decision-making should you reach for an agent.

Even then, start with one. The best AI systems are the simplest ones that reliably solve the problem. That usually means starting with workflows.

When a Workflow Is the Right Answer

Workflows are the right answer when your steps are known and stable. If the process is largely the same each time, regardless of input, a workflow is almost always the best choice.

Workflows win because they are predictable. They are easy to test because you can write unit tests for each step. They are easy to debug because you can trace exactly what happened when something goes wrong.

They are also cheap because you are not burning tokens on the model, figuring out what to do next.

Consider a support ticket system. A ticket comes in. You classify it.

You route it to the right team. You draft a response from templates and context. You validate it against the policy.

Finally, you send it. Each step might involve an LLM call, but the model does not need to decide whether to classify before routing. That is always the order.

Building this as an agent adds overhead without adding capability.

Do not underestimate workflows. They are not limited to simple sequential chains. They can include routing to pick different models based on input.

They can use parallel execution with majority voting to aggregate answers. They can also use generator-evaluator loops where one LLM generates and another validates until quality criteria are met. They can even leverage tools in designs like the orchestrator-worker. These patterns handle complex tasks without any agent overhead.

If you can write down the exact sequence of steps in advance, like a recipe, it is a workflow.

When a Single Agent with Tools Wins

Sometimes the order of work is not fixed. You genuinely cannot write down the steps in advance. This happens when the path changes depending on what you discover along the way.

Maybe the first API call fails, and you need to try an alternative. Maybe the retrieved data is incomplete, and you need clarification. This is what agents handle well.

When is an agent worth the risk? Anthropic offers a useful framework. Agents make sense when the task is complex enough to need autonomous decisions and delivers real value.

Critically, the cost of errors and the cost of discovering those errors must be low. This is why AI coding agents are great. A human reviews the code before production, so mistakes are cheap to fix.

A purchasing agent who accidentally buys the wrong hardware makes an expensive error. You must match your architecture to your error tolerance [3].

The rule is to always start with one agent. A single agent with tools works best when tasks are tightly coupled and mostly sequential. It works well when global context matters, meaning step one affects step five.

It is also ideal when you need fewer than twenty tools and face strict budget or latency constraints.

Take a marketing content platform from Louis-François’s client work at Towards AI. The client wanted AI-assisted content generation for emails, text messages, and promotional materials. Their initial specification called for a multi-agent setup with over a dozen specialized agents.

They wanted an orchestrator, a request analyzer, a content generator, and many others. On paper, it looked clean with specialists doing specialist work.

A single agent was the right call. The tasks were tightly coupled and sequential. The template choice affects the content.

Personalization depends on both content and contact data. Splitting this across multiple decision makers creates information silos and handoff errors. They did not need parallelism.

The flow was to plan, generate, validate, and fix if needed.

The key insight is that tools can be smart. A tool can have its own system prompt and use a different model. The validation tool can use its own LLM with instructions to catch errors.

The text message tool can treat character limits as deterministic engineering constraints instead of prompting problems. You get specialists, but you keep one brain to maintain context and make final decisions.

This results in a system that is faster to build, cheaper to run, and easier to debug. You get the same capabilities without the coordination overhead.

The Tool Count Problem: When One Agent Isn’t Enough

As your tool list grows, tool selection gets harder. This is one of the main ways agent systems quietly break down. It is also one of the clearest signals that splitting into multiple agents might be worth it.

Every tool has a name, description, and schema that the model needs in context to use correctly. The more tools you add, the more of your context budget you burn before the agent even starts thinking about the actual task. You also have to add system instructions, a few-shot examples, retrieved documents, and conversation history on top of that.

A single agent tends to work best with roughly 10 to 20 tools. Past that threshold, tool selection degrades. The agent has to choose among too many options in an already packed context.

This mechanism is known as context rot. LLM performance measurably degrades as context grows, well before hitting the advertised limit. Two forces drive this issue.

First, more context means more noise competing for the model’s attention. Second, models suffer from loss in the middle. They tend to attend more to the beginning and end of their context, underweighting information in the middle.

As your tool schemas and instructions pile up, the model gets worse at picking the right tool.

Managing context can reduce history and retrieved content, but not the tool schema load. Those definitions must always be there. The only approach that actually reduces how many tool definitions the model sees per call is splitting across agents.

If one agent sees only email tools and another only sees validation tools, each call stays smaller. Tool selection gets easier. Once you split tools across agents to keep calls small, you enter multi-agent territory.

When Multi-Agent Is Actually the Right Call

Specific reasons justify multiple agents, not because the architecture sounds impressive. There are four legitimate reasons to go multi-agent. First, you need true parallelism where tasks are genuinely independent and run simultaneously.

Second, you face context overload where instructions and tools degrade performance. Third, you need modularity to connect with third-party agent systems you do not control. Fourth, you have hard separation requirements like security boundaries or sensitive data handling.

Consider the professional article generation system that Louis-François and I built as one of the projects for our Agentic AI Engineering course. We started with a single agent for research and writing but had to pivot because the two phases have fundamentally different needs.

The research phase is exploratory and dynamic. It needs flexibility and broad tool access across web search, video transcription, and document processing. The agent searches, reads, pivots based on what it finds, and iterates based on human feedback.

The writing phase is constrained and deterministic. It needs focused constraints, consistent style enforcement, and iterative refinement against fixed rubrics.

These agents communicate through explicit artifacts. The research agent produces a structured markdown file that the writer agent consumes as context. There is no complex runtime orchestration.

It is just a sequential handoff with a clear contract between them. Each agent has its own optimized context without the bloat of carrying the other’s tools.

If you do go multi-agent, we recommend the plan-and-execute combined with the orchestrator-worker pattern. You do not want everyone talking to everyone. One orchestrator maintains the main context and delegates specific tasks to worker agents.

Then, it synthesizes the results. This prevents the information silos that kill multi-agent systems.

Multi-agent systems can simplify individual contexts and enable specialization. However, they increase coordination costs. You will face more token usage, added latency, more failure points, and handoff complexity.

Only accept those costs when you hit a real constraint that simpler architectures cannot solve.

To Wrap Up

To build reliable AI applications, you must stay as far left on the complexity spectrum as possible while still solving your problem.

Keep these key takeaways in mind:

• Not every LLM application is an agent, and not every tool is an agent.

• Always start with workflows because they are predictable, cheap, and testable.

• Use one agent when the path cannot be predetermined, but keep the tool count manageable.

• Move to multi-agent architectures only when you hit a real constraint like true parallelism or context overload.

Each step right on the spectrum increases cost, latency, and debugging complexity. The simplest system that reliably solves the problem is always the best system.

💡 If you want a step-by-step framework to help you decide what architecture to pick for your next project, Louis-François and the Towards AI team put together a free cheatsheet that walks you through the decision process from workflows to multi-agent systems.

What’s your opinion? Do you agree, disagree, or is there something I missed?

Leave a comment

Enjoyed the article? The most sincere compliment is to share our work.

Share

Go Deeper

Go from zero to production-grade AI agents with the Agentic AI Engineering self-paced course. Built in partnership with Towards AI.

Across 34 lessons (articles, videos, and a lot of code), you’ll design, build, evaluate, and deploy production-grade AI agents end to end. By the final lesson, you’ll have built a multi-agent system and a capstone project where you apply everything you’ve learned on your own.

Three portfolio projects and a certificate to showcase in interviews. Plus a Discord community where you have direct access to other industry experts and me.

Rated 4.9/5 ⭐️ by 300+ students saying “Every AI Engineer needs a course like this.”

Learn more

Not ready to commit? We also prepared a free 6-day email course to reveal the 6 critical mistakes that silently destroy agentic systems. Get the free email course.

Images

If not otherwise stated, all images are created by the author.

Welcome to the AI Evals & Observability series: A 7-part journey from shipping AI apps to systematically improving them. Made by busy people. For busy people.

AI Evals is the topic most AI engineers know they should invest in, but do not know where to start. I remember struggling with this myself.

I did not know how to properly integrate evals into my app until I understood there are three core layers: optimization during development, regression testing before merging, and production monitoring on live traffic. Once that clicked, everything else fell into place.

I did not know how to build LLM judges and evaluators that I could actually trust and use. Every guide I found either hand-waved the details or dumped a generic “helpfulness” metric and moved on. Instead, I needed evaluators grounded in my actual business requirements.

I did not know how to gather custom datasets without wasting too much time. I tried generating hundreds of synthetic test cases up front, but the real unlock came from learning how to organically grow a high-quality dataset from production data, starting small and letting the error-analysis flywheel do the heavy lifting.

The information was scattered across blog posts, talks, and vendor docs. Most of it focused on isolated techniques without showing how everything connects. I built this series as the structured, end-to-end guide I wish I had.

This 7-lesson series breaks it all down from first principles. By the end, you will know how to integrate AI evaluations that actually track and improve your product's performance. No vibe checking required.

The series follows a natural progression. You start by understanding where evals fit. Then, you build the dataset.

Next, you design and validate the evaluators. Finally, you handle specialized domains like RAG and see how it all works in production.

You can read front-to-back for the full journey. Alternatively, jump to the lesson that matches your current pain point. Each lesson stands on its own but references the others.

Without more yada, yada, here are the 7 lessons of the series:
(Scroll down to find more about each lesson individually.)

1. Integrating AI Evals Into Your AI App

2. Build an AI Evals Dataset from Scratch

3. Generate Synthetic Datasets for AI Evals

4. How to Design Evaluators

5. How to Evaluate the Evaluator

6. RAG Evaluation: The Only 6 Metrics You Need

7. Lessons from 6 Months of Evals on a Production AI Companion

Everything is completely free, without any hidden costs, thanks to our sponsor, Opik ↓

Opik: Open-Source LLMOps Platform (Sponsored)

This AI Evals & Observability series is brought to you by Opik, the LLMOps open-source platform used by Uber, Etsy, Netflix, and more.

We use Opik daily across our courses and AI products. Not just for observability, but as our end-to-end evaluation harness, all from the same platform.

This series teaches you how to build evals from scratch (custom datasets, LLM judges, optimization loops, and production monitoring), while Opik gives you the platform to run everything at scale.

Here is how we use it:

• Custom LLM judges: Build evaluators by defining your criteria, adding a few-shot examples, and running them across hundreds of traces automatically.

• Run experiments, compare results: Test different prompts, models, or parameters from your AI app side by side. Opik scores each variant with your evaluators and shows you which one wins.

• Plug evaluators into production: The same LLM judges you design for offline testing run on live traces too. Set up alarms when scores drop below your threshold so you catch regressions before users do.

Opik is fully open-source and works with custom code and with every popular AI framework or tool (including OpenClaw). You can also use the managed version for free (with 25K spans/month on their generous free tier):

Try Opik for free

↓ Now, let’s move back to the article.

Lesson 1: Integrating AI Evals Into Your AI App

To build a reliable system, you first need to know where evaluation fits into the development lifecycle.

Most teams start by “vibe checking” their AI app. They manually test a few inputs and eyeball whether the outputs look right. That works for the first version.

But the moment you start adding features, onboarding real users, or trying to improve existing capabilities, vibe checking collapses. This first article gives you the holistic map of where AI Evals fit, so you never feel lost again.

Here is what you will learn:

• The three core scenarios where evals matter: optimization during development, regression testing before merging, and production monitoring on live traffic.

• The difference between guardrails and evaluators. Confusing them leads to gaps in your system.

• The minimum viable tech stack required to start: a custom annotation tool and an LLMOps platform.

Go to Lesson 1

Lesson 2: Build an AI Evals Dataset from Scratch

Once you understand where evals fit, the next step is gathering the data required to measure performance.

You cannot evaluate what you cannot measure. You cannot measure without data. Most teams either skip this step entirely or fire off a generic prompt to create 100 test cases and call it done.

This article teaches the error analysis framework. It is a practical flywheel that turns 20-50 real production traces into a growing, high-quality evals dataset.

Here is what you will learn:

• The error analysis flywheel: sample traces, label manually, build evaluators iteratively, perform error analysis, and create specialized evaluators.

• Why one “benevolent dictator” should own labeling consistency across your team.

• How to graduate from generic to specialized evaluators as your understanding deepens.

Go to Lesson 2

Lesson 3: Generate Synthetic Datasets for AI Evals

Production traces alone have limits. You need traffic to get data, and that traffic rarely covers every scenario. What about before you have users?

What about rare failure modes you have never seen in production? Yet! Synthetic data solves the cold start problem and fills coverage gaps.

Here is what you will learn:

• Why you should generate only inputs, not outputs, and let your real app produce the outputs.

• How to think in dimensions like persona, feature, scenario, and input modality to avoid mode collapse.

• Tester agents for simulating multi-turn conversations.

• The reverse workflow for RAG: generate questions from your knowledge base, not the other way around.

Go to Lesson 3

Lesson 4: How to Design Evaluators

You have the dataset. Now you need evaluators who can actually tell you whether your app is working. This is where most teams make their biggest mistake.

They grab a generic helpfulness metric off the shelf and call it done. This article teaches you how to design evaluators grounded in your actual business requirements.

Here is what you will learn:

• The evaluation harness: the infrastructure that automates running evaluators across your dataset.

• When to use fast, deterministic code-based evaluators versus flexible, nuanced LLM judges.

• Common design mistakes

• Advanced designs for multi-turn conversations and agentic workflows.

Go to Lesson 4

Lesson 5: How to Evaluate the Evaluator

You built an evaluator. It says everything is great. But is it?

An evaluator that validates every output is worse than no evaluator at all. It gives you false confidence. This article teaches you how to validate your evaluator against human judgment and close the gap when they disagree.

Here is what you will learn:

• The iterative refinement loop: measure alignment, diagnose disagreements, adjust few-shot examples, and re-measure.

• Dealing with non-determinism: why LLM judges give different answers on the same input, and how to stabilize them.

Go to Lesson 5

Lesson 6: RAG Evaluation: The Only 6 Metrics You Need

After mastering general evaluators, you can apply these principles to specific architectures like RAG.

RAG evaluation feels overwhelming because everyone proposes different metrics. But it does not have to be complicated. This article proves that there are exactly three variables in any RAG system: Question, Context, and Answer.

There are exactly six possible relationships between them. That is it. Every RAG metric maps to one of these six relationships.

Here is what you will learn:

• The three RAG variables and six exhaustive relationships.

• Tier 1: Retrieval metrics. If retrieval is broken, nothing else matters.

• Tier 2: The three core RAG metrics you always need.

• Tier 3: When core metrics cannot explain the failure.

Go to Lesson 6

Lesson 7: Lessons from 6 Months of Evals on a Production AI Companion

Theory and isolated metrics are useful. But the ultimate test is running this entire system on live user traffic.

The first six articles teach you how to build the system. This final article shows you what it looks like after six months of running it in production.

Written as a guest post by , Senior Data Engineer at Workpath, it shares the real lessons. We cover what worked, what failed, and what they wish they had known from the start.

Here is what you will learn:

• The three observability problems most teams hit: falling for generic metrics, skipping manual annotation, and not treating AI agents as data products.

• How to use Opik’s architecture, including traces, spans, threads, and prompt versioning, for production monitoring and evals.

• How to reverse-engineer evaluation criteria from real traces instead of guessing upfront.

Go to Lesson 7

How to Take the Course?

After completing these seven articles, you will have the complete mental model for AI Evals. You will understand everything from strategy to production.

As the course is 100% free, with no hidden costs or registration required, taking it is a no-brainer.

Each lesson is a free article hosted on the Decoding AI Magazine.

Just open each lesson in the order provided by us, and you are good to go:

1. Integrating AI Evals Into Your AI App

2. Build an AI Evals Dataset from Scratch

3. Generate Synthetic Datasets for AI Evals

4. How to Design Evaluators

5. How to Evaluate the Evaluator

6. RAG Evaluation: The Only 6 Metrics You Need

7. Lessons from 6 Months of Evals on a Production AI Companion

Each lesson will guide you through the required steps.

Enjoy!

Now What?

After completing these lessons, if you want the information to stick, you have to put everything into practice by building a cool project!

I am sorry to say there is no other way to make learning worthwhile. Pick one problem and get your hands dirty with a project.

💡 Want to share your work on my socials with my 140k+ audience? If you build a project you are excited about, I will be too. Trust me! I love seeing people build cool stuff. To share it, you can contact me here.

See you next Tuesday.

Paul Iusztin

What’s your opinion? Do you agree, disagree, or is there something I missed?

Leave a comment

Enjoyed the article? The most sincere compliment is to share our work.

Share

Go Deeper

Go from zero to production-grade AI agents with the Agentic AI Engineering self-paced course. Built in partnership with Towards AI.

Across 34 lessons (articles, videos, and a lot of code), you’ll design, build, evaluate, and deploy production-grade AI agents end to end. By the final lesson, you’ll have built a multi-agent system and a capstone project where you apply everything you've learned on your own.

Three portfolio projects and a certificate to showcase in interviews. Plus a Discord community where you have direct access to other industry experts and me.

Rated 4.9/5 ⭐️ by 300+ students — “Every AI Engineer needs a course like this.”

Learn more

Not ready to commit? We also prepared a free 6-day email course to reveal the 6 critical mistakes that silently destroy agentic systems. Get the free email course.

Thanks again to Opik for sponsoring the series and keeping it free!

If you want to monitor, evaluate and optimize your AI workflows and agents:

Try Opik for free

Images

If not otherwise stated, all images are created by the author.

Most agents fail silently. They work well in demos but drift unpredictably in production. Costs spike with no clear explanation.

Behavior becomes erratic, and every release feels risky. Ultimately, teams end up stuck in PoC purgatory, unable to ship, debug, or trust their own system.

The root cause is almost never the model. It is subtle system design mistakes that individually look small but compound into production disasters.

To fix this, together with , we created a diagnostic framework for six specific mistakes that cause agentic systems to break in production. Each has a clear problem, a reason why it happens, and a proven fix. Once you know what to look for, you can trace most production failures back to one of these patterns.

The first and most common failure starts right at the input level, where engineers mishandle the context window.

Mistake #1: Treating the Context Window as an Afterthought

When something breaks, the instinct is to add more context. Engineers add more rules, more history, more tools, and more examples. The assumption is that if the model sees everything, it will behave better.

But this turns the context window into a dumping ground instead of a carefully scoped working memory. As the context grows, the model starts to ignore instructions and apply constraints inconsistently. It hallucinates more and drifts across runs.

Latency spikes and costs compound. This is the lost in the middle problem. Many teams respond by splitting one giant prompt into dozens of smaller ones.

But that introduces its own problems, such as more LLM calls, higher latency, and harder debugging.

💡 Treat the context window as a scarce resource.

Every LLM call should have one clearly scoped job. You must curate context aggressively by selecting, compressing, and pruning before every call. Move persistence into a memory layer.

The context window holds only what matters for the next decision, and everything else lives in memory, which you write to and read from continuously.

As a rule of thumb, start with a single prompt. If it works, stop. If it fails, do not jump to agents.

Introduce a small number of specialized steps and tune until you hit the balance. Context engineering is about deliberate selection.

Once the context window is secure, the next trap is overengineering the architecture before the problem demands it.

Mistake #2: Starting with Complicated Solutions

You have a clear problem, so you immediately reach for multi-agent architectures or heavy frameworks. You build RAG pipelines, hybrid retrieval, multiple databases, or adopt new protocols like MCP. You do this not because the problem demands it, but because it feels like the right way to build serious AI.

Every layer adds a hidden tax. You get more dependencies, higher latency, higher costs, and harder debugging. Complexity compounds operational pain.

Teams end up spending months building infrastructure and shipping nothing.

At our startup, ZTRON, we built a multi-index RAG system. We had OCR pipelines, separate embedding pipelines, hybrid retrieval, and agentic RAG loops.

It worked, but simple queries took 10 to 15 seconds. Costs climbed, and debugging was a nightmare.

When we finally asked if we actually needed all this, the answer was no. Our data fit within modern context windows. We replaced agentic RAG with cache-augmented generation (CAG) for most workflows.

This gave us fewer LLM calls, lower latency, fewer errors, and an easier system to debug.

Start with the simplest solution that could work. Prove the core task works first. Only add memory, tools, retrieval, or multiple agents when the problem demands it.

Production-grade AI is built by engineers who ship simple systems first and scale complexity intentionally.

Earning complexity often means realizing that you do not need an agent at all, which brings us to the third mistake.

Mistake #3: Building Agents When a Workflow Will Do

Predictable tasks like data ingestion, summarization, or report generation need predictable execution. That is a workflow. Open-ended tasks like deep research or dynamic decision-making under uncertainty may need autonomy.

Agents handle these open-ended scenarios. Most teams treat predictable problems as if they need agents. When you use an agent for a structured task, you pay for autonomy you do not need.

You get unpredictable behavior, variable latency, higher token usage, and inconsistent outputs. The system works 80% of the time and fails when it matters most.

Workflows and agents are not binary choices. They sit on a spectrum known as the autonomy slider. More autonomy buys flexibility but costs predictability, cost control, and debuggability.

You must set the slider intentionally.

Adopt a workflow-first approach. Start with prompt chaining, routing, parallelization, or an orchestrator-worker pattern. Introduce agents only when the system must autonomously plan, explore unknown paths, or recover from failures dynamically.

For vertical AI agents, use a hybrid approach. Route known patterns to workflows and open-ended requests to agents.

Whether you use a workflow or an agent, you must handle the data they produce, which exposes a flaw in how engineers process outputs.

Mistake #4: Fragile Parsing of LLM Outputs

You ask the model for something structured, and it responds with something that looks structured. You parse it with regex, string splitting, or custom logic. It works in staging.

Then one day, a missing comma or different bullet style crashes production. LLMs are non-deterministic. Even with identical prompts, output can drift due to context changes, model updates, or variations in tool outputs.

Fragile parsing is a time bomb. Many teams respond by prompting the model to output JSON. That is better than free-form text, but it still is not a contract.

You still get missing keys, wrong types, and drifting nested fields.

Stop treating LLM outputs like text and treating them like data. Define a schema, enforce it at generation time, validate at runtime, and fail fast when wrong. Use Pydantic as the bridge between probabilistic generation and deterministic code.

But only use structured outputs when structure is required. If you only need a plain string, accept a string and keep schemas shallow and minimal.

If you have secured your context, simplified your architecture, chosen the right autonomy, and enforced output schemas, you are ready to build an agent. However, many teams still fail by omitting actual planning from their loops.

Mistake #5: Forgetting Agents Need Planning

You give a model tools, let it pick one, feed the tool output back, and repeat. At a glance, it looks agentic, but it is just a workflow with randomness. The system is reacting to the last tool output, not driving toward a goal.

Without embedded planning, the loop cannot decompose tasks into meaningful steps. It cannot evaluate progress or choose next actions intentionally. The result is random behavior, unnecessary tool calls, infinite loops, and shallow reasoning.

Copying ReAct or Plan-and-Execute from blog posts without adapting them to your domain makes it worse.

You must embed planning into the loop. Before calling a tool, require a reasoning step. Ask what the goal is, what the next best action is, and what evidence you need.

Add progress checks and stop conditions like max steps, token budgets, and escalation when stuck. Make planning use-case specific, because generic ReAct is not a product. Tailor planning to your tools, data, constraints, and failure modes.

Even a well-planned agent will degrade over time if you do not measure its performance continuously.

Mistake #6: Not Starting with AI Evals from Day Zero

You build features without tracking how well your AI behaves. You have no tests, no evaluation metrics, and no defined success criteria. Every new feature is a gamble, and teams silently ship regressions.

AI systems do not fail all at once. They decay. A prompt change, a new tool, or a model upgrade causes subtle behavior shifts.

Without evals, nobody can answer whether a change made the system better or worse. Teams get stuck relying on vibe evals, which are manual, gut-feel testing that does not scale. Many teams think they are doing evals, but rely on generic scores like helpfulness or 1-5 star scales.

A score of 3.7 helpfulness tells you nothing about what to fix.

Use evals as your north star. Define task-specific, binary metrics tied to real system behavior and business requirements from day one. Use evals to drive the optimization flywheel.

Integrate evals into your development workflow to catch regressions before users do.

Recognizing these six mistakes is the first step to escaping PoC purgatory.

Conclusion

These six mistakes are not exotic edge cases. They are the exact patterns that repeatedly break real agentic systems. Individually, they look small, but in production, they compound into disasters.

Each of these mistakes deserves a deeper breakdown with real examples and production-tested fixes. That is why we turned them into a free 6-day email course. We cover one mistake per day, with the exact patterns and solutions we use in production.

💡 If you want the complete breakdown, sign up here.

Otherwise, see you next Tuesday.

Paul Iusztin

What’s your opinion? Do you agree, disagree, or is there something I missed?

Leave a comment

Enjoyed the article? The most sincere compliment is to share our work.

Share

Go Deeper

Go from zero to production-grade AI agents with the Agentic AI Engineering self-paced course. Built in partnership with Towards AI.

Across 34 lessons (articles, videos, and a lot of code), you’ll design, build, evaluate, and deploy production-grade AI agents end to end. By the final lesson, you’ll have built a multi-agent system and a capstone project where you apply everything you’ve learned on your own.

Three portfolio projects and a certificate to showcase in interviews. Plus a Discord community where you have direct access to other industry experts and me.

Rated 4.9/5 ⭐️ by 300+ students saying “Every AI Engineer needs a course like this.”

Learn more

Not ready to commit? We also prepared a free 6-day email course to reveal the 6 critical mistakes that silently destroy agentic systems. Get the free email course.

Images

If not otherwise stated, all images are created by the author.

🧐 Everyone says you need AI evals. Few explain how to actually build them and answer questions such as…

How do we avoid creating evals that waste our time and resources? How do we build datasets and design evaluators that matter? How do we adapt them for RAG? ...and most importantly, how do we stop “vibe checking” and leverage evals to actually track and optimize our app?

This 7-article series breaks it all down from first principles:

1. Integrating AI Evals Into Your AI App

2. Build an AI Evals Dataset from Scratch

3. Generate Synthetic Datasets for AI Evals

4. How to Design Evaluators

5. How to Evaluate the Evaluator

6. RAG Evaluation: The Only 6 Metrics You Need ← You are here

7. Lessons from 6 Months of Evals on a Production AI Companion

By the end, you’ll know how to integrate AI evals that actually track and improve the performance of your AI product. No vibe checking required!

Let’s get started.

RAG Evaluation: The Only 6 Metrics You Need

In our previous article, we covered how to validate your AI judges. We measured agreement with human judgment and iterated until alignment was high. Thus, you can now deploy with confidence.

However, a specialized challenge exists that general-purpose grading tools do not fully address. Evaluating RAG systems introduces a third variable, specifically the retrieved context. With this new element comes a distinct set of failure modes requiring their own metrics.

I am currently building a financial personal assistant at the stealth AI startup I work for. The application runs heavily on RAG. It pulls financial data from Postgres and integrates with external services such as email, Customer Relationship Management (CRM) tools, and cloud drives.

When it came time to evaluate the system, building the dataset proved harder than choosing metrics. Fortunately, we had a domain expert on the team who manually tested the application from the start. Therefore, we translated all of that Quality Assurance (QA) work into our AI evals collection using the error analysis workflow from Article 2.

Evaluating RAG systems introduces a unique difficulty. Each data sample required the correct context to be loaded into the database. We solved this by coupling every test case with a Postgres SQL export.

This file contained documents, chunks, embeddings, and metadata. We injected it directly into the storage system. This effectively created a cache that bypassed the ingestion pipeline during evals.

Once the data was in place, implementing the core RAG metrics became straightforward. We used tools like Opik and foundational models like Gemini Pro as the LLM judge. We had the context, the query, and the answer, which is everything you need.

What surprised me was that not every capability needed this level of dissection. For our report generation feature, we expect an exact format with specific values pulled from the storage. Checking the final document against a ground truth served as a better proxy than tracing every retrieval step.

Sometimes assessing the destination matters more than checking the route.

RAG evaluation feels needlessly complex. Vendors have an incentive to make it difficult. Every framework ships with many metrics and a dashboard, making you feel like you need a PhD to know if your system works.

Underneath all the complexity, RAG systems possess exactly three core components. These are the Question (Q), the retrieved Context (C), and the generated Answer (A). Furthermore, with these elements, there are exactly six possible relationships between them. When your RAG system fails, it breaks along one of these six relationships every single time.

The beauty of this framework is its exhaustive nature. There are no hidden variables.

You do not always need to evaluate all six individually. For core conversational features, you need the primary metrics because there are many silent failure modes. However, for structured output tasks, an end-to-end check against expected results can be sufficient.

Here is what you will learn in this article:

• The only six relationships that exist in a RAG system.

• How to evaluate your retrieval step before looking at generation.

• The three core metrics every RAG application needs.

• Advanced metrics for diagnosing subtle hallucinations.

• How to match evaluation frequency and strictness to your domain.

• How to collect and prepare the data your evaluators need.

Before digging into the article, a quick word from our sponsor, Opik. ↓

Opik: Open-Source LLMOps Platform (Sponsored)

This AI Evals & Observability series is brought to you by Opik, the LLMOps open-source platform used by Uber, Etsy, Netflix, and more.

We use Opik daily across our courses and AI products. Not just for observability, but to design and run the exact RAG evaluators this article teaches. All from the same platform.

This article shows you how to evaluate RAG systems. Opik gives you the harness to run those evaluations at scale. Here is how we use it:

• Custom LLM judges with rubrics — Build the evaluators this article describes: define your criteria, add few-shot examples, and run them across hundreds of traces automatically.

• Run experiments, compare results — Test different prompts, models, or configurations side by side. Opik scores each variant with your evaluators and shows you which one wins.

• Plug evaluators into production — The same LLM judges you design for testing run on live traces too. Set up alarms when scores drop below your threshold so you catch regressions before users do.

Opik is fully open-source and works with custom code or most AI frameworks. You can also use the managed version for free (with 25K spans/month on their generous free tier):

Try Opik for free

↓ Now, let’s move back to the article.

The Only 6 RAG Evaluation Metrics That Can Exist

Jason Liu properly articulated the framework I am about to walk you through [1]. Since I wrote the LLM Engineer’s Handbook two years ago, I have watched many RAG evaluation tools emerge. They overcomplicate everything with proprietary metric suites.

Through all of that, I already internalized that only three variables matter in any RAG system. Testing the combinations between them is the only thing you should actually do. Jason Liu gave a clean, formal articulation to what I had in mind.

He nailed the structure and deserves the recognition for that.

Every RAG system has three variables. We define Q as the user’s question, C as the retrieved context, and A representing the generated answer. Thus, we use the notation X|Y to mean the quality of X given Y.

There are exactly six relationships between these variables:

1. C|Q (Context Relevance) asks if the retrieved context addresses the question. This measures your retriever, because if it pulls irrelevant passages, the generator cannot fix the issue.

2. A|C (Faithfulness) checks if the answer sticks to what is in the context. This measures your generator to see if the model hallucinated or stayed grounded in the documents.

3. A|Q (Answer Relevance) verifies if the response actually addresses the prompt. This is the end-to-end user experience metric. Even if the context is good and the reply is faithful, it must help the person asking.

4. C|A (Context Support) ensures the retrieved text contains everything needed to support every claim in the answer. This checks if the provided information was sufficient.

5. Q|C (Question Answerability) evaluates if the prompt can even be resolved with this context. This determines whether the system should attempt to reply at all.

6. Q|A (Self-Containment) asks if someone can infer the original question from reading the answer alone. This measures whether the output provides enough background to stand on its own.

This framework is exhaustive. Three components produce exactly six conditional relationships. There are no hidden factors.

Therefore, when your RAG system fails, one of these six metrics is broken.

Not all six relationships matter equally in every context. We organize them into three tiers. Let us start with retrieval metrics as the prerequisite foundation.

Tier 1: If Retrieval Is Broken, Nothing Else Matters

RAG is first and foremost a retrieval problem. If the search mechanism does not retrieve the right documents, nothing downstream can save you. The generator will either hallucinate or produce irrelevant answers based on whatever junk it received.

Before evaluating any of the six RAG relationships, you need to know if your retriever even works. You can use classic information retrieval metrics that measure how well you find relevant documents before generation starts. They are fast to compute and do not require LLM judges.

Hence, these measurements give quick feedback for tuning your retriever.

You must establish ground-truth labels to compute these metrics. For each query, you must know which text blocks are actually relevant. You can build this dataset using the reverse workflow presented in depth in Article 3.

As a quick recap, you start from your knowledge base of document chunks. Then, based on a set of closely related chunks, you generate realistic questions that can only be answered using that unique set of chunks.

Because the prompt derives from the source material, you know exactly which segment should be retrieved. This gives you a perfectly aligned ground-truth triplet: (question, answer, context). Thus, it becomes straightforward to check whether your search tool actually surfaces the right information.

There are four main metrics. Precision@K measures the fraction of the top K retrieved chunks that are actually relevant. If your retriever returns 5 chunks but only 2 are useful, your precision is 40%. Recall@K asks: of all the relevant chunks that exist in your entire corpus, how many did your retriever actually find in the top K? If the database has 4 chunks that could answer the question but you only retrieved 2 of them, your recall is 50%.

In addition, Mean Average Precision (MAP@K) averages precision across multiple queries, rewarding retrievers that consistently rank relevant chunks early. It works by computing precision at every position where a relevant item appears, then averaging those values. Here is a step-by-step example where the truly relevant items are A and C:

Average Precision for this query = (1.0 + 0.66) / 2 = 0.83. We only average the precision values at positions where a relevant item appeared (ranks 1 and 3). MAP@K then takes this score and averages it across all your queries.

Finally, Mean Reciprocal Rank (MRR@K) focuses on the position of the first relevant match. If the first relevant chunk appears at position 3, the reciprocal rank is 1/3; if it appears at position 1, it is 1/1. Higher is better.

Use these for daily development. These indicators are great for tuning embeddings and chunk sizes, while also being perfect for A/B testing retrieval strategies. No LLM is needed, making the process cheap and fast.

These numbers tell you if the search phase works, as illustrated in Image 3. The six RAG relationships tell you if the whole system functions properly, meaning you need both.

With the retrieval confirmed working, you can evaluate the generation step. Let us look at the three core RAG relationships that every system needs.

Tier 2: The Three RAG Metrics You Always Need

These three metrics directly assess how well your RAG system functions. Most evaluation frameworks prioritize these specific measurements. They map to the three most critical of the six relationships.

First, we have Context Relevance (C|Q). This checks if the retrieved text actually addresses the prompt’s information needs. Therefore, it measures your search component similar to the metrics from Tier 1, but only looking at the dynamics between the context and question, without any ground truth.

Suppose we have a query about recent payouts from Q4. A good example is when the retrieved data contains the user’s dividend payment records from Q4, which passes. On the other side, a bad scenario occurs when the system returns general information about how these distributions work and their tax implications.

This represents the most common RAG failure mode. In our financial assistant, this often happened when the search tool pulled educational content instead of actual account data.

Second, we have Faithfulness (A|C). This asks if the reply restricts itself to claims that can be verified from the provided text. Hence, it measures whether your generator hallucinates or not.

In our use case, a good example is when the source contains a CRM record showing a client meeting scheduled for portfolio rebalancing. If the response states exactly that, it passes. A bad example happens when the model adds hallucinated agenda items like tax-loss harvesting strategies, resulting in a failure.

Third, we have Answer Relevance (A|Q). This checks if the output directly addresses the specific query from the prompt. This serves as the end-to-end user experience metric.

A good example is when a person asks how much their investments grew last month. The reply provides the specific percentage change and absolute dollar amount. A bad scenario is when the text discusses general market performance without mentioning the actual account.

We measure all three metrics using LLM judges as designed in Article 4 and validated in Article 5.

These three metrics cover the most common failure modes. For specific domains and failure cases, we have to dig deeper into the next 3 metrics.

Tier 3: When the Core Metrics Can’t Explain the Failure

The last three metrics provide deeper diagnostic insights usually required in sensitive domains or use cases.

First, we have Context Support (C|A). This checks if the retrieved context contains all the information needed to fully back every claim in the response. While this sounds similar to Faithfulness (A|C), the direction is different. Faithfulness asks: “did the answer deviate from the context?” , where you look at the answer and check if it introduced claims that aren’t there. Context Support asks: “was the context sufficient to support the answer?”, where you look at the context and check if it actually contains everything the answer needs.

Here is a concrete example. Suppose the answer says your total Q4 dividend income was 2,340 across 5 holdings, with the largest payout from MSFT at 890. Now look at the context: it only contains the total dividend amount of $2,340. The per-holding breakdown is nowhere in the retrieved documents. The context was insufficient. It had the total but not the details. The LLM produced a plausible breakdown, but the context could not support it. This is low-context support.

Second, we have Question Answerability (Q|C). This asks if the user's question can even be resolved with the given information.

Suppose the user asks about crypto portfolio performance, but the retrieved documents only contain equity data. This makes the request unanswerable. The system should refuse rather than guess. This metric is important when you want to validate that your agent answers with “I don’t know” instead of confidently hallucinating an answer due to insufficient context.

In our financial assistant, this was important because some queries can only be resolved if the agent has permissions to access the right external tool first.

Third, we have Self-Containment (Q|A). This checks if someone can infer the original prompt from the reply alone.

A response stating your portfolio’s return is 12.4% stands alone. A reply stating just 12.4% does not. Prioritize this metric when outputs are forwarded via email, logged in CRM notes, or read without the original conversation.

You now know what to measure at each tier. Two questions remain. How often should you run each one? Which metrics deserve the most attention for your specific domain?

Matching Frequency and Strictness to Your Domain

Each tier maps to a different running frequency depending on how fast and cheap you can run the evaluations. It also depends on their overall impact on the system.

Start with Tier 1 on a daily basis. Implement fast retrieval metrics for everyday development and to tune your retrieval component. These are the cheapest to execute as they do not require LLM judges.

Furthermore, they provide quick feedback cycles. Use them for the improvement flywheel with synthetic data from day zero, focusing on these basic indicators before moving to more complex approaches.

Move to Tier 2 on a weekly basis. Implement the three primary RAG connections. These core metrics directly assess how well your system functions.

Use LLM-based grading for a more nuanced assessment of these interactions.

Incorporate Tier 3 on a monthly basis. Introduce advanced metrics when you need deeper insights. Run a full evaluation to identify prompts that the application should not be answering.

Here, we focused only on RAG-related measurements. However, this actually applies to any type of AI evals layer. You could implement Tier 1 checks in your CI/CD pipeline to execute on each commit.

You can trigger Tier 2 evaluations manually before merging your code from your feature branch. Finally, manually run Tier 3 metrics before major releases and strategic decisions.

There is another dimension to consider when choosing metrics for your use case, which is the good old domain.

Different domains require emphasis on distinct indicators. What matters most depends on the severity of the use case.

High-severity domains include finance, medical, and legal applications. In these fields, Faithfulness (A|C) and Context Support (C|A) are non-negotiable because every claim must be traceable. Answerability (Q|C) is also critical, meaning the application must refuse rather than guess.

Thus, you want precision over recall, which is the exact profile we use for our financial assistant.

Medium severity domains include customer support and technical documents. Answer Relevance (A|Q) leads here, as the output must be helpful and correct. Answerability (Q|C) helps you know when to hand off to a human, and you generally want recall over precision in retrieval.

Low-severity domains include research, writing, and content generation, where synthesis and creative reframing are expected. Context Relevance (C|Q) and Answer Relevance (A|Q) is primary, while Faithfulness (A|C) thresholds remain lower. The generator is supposed to add value beyond the raw text.

Therefore, you want high recall in the search phase to cast a wide net across sources.

You know what to measure, when, and what to prioritize. None of this works without the right data and infrastructure. Let us explore how to build the evaluation harness.

Building the RAG Evaluation Harness

RAG evaluation requires inputs, outputs, and the retrieved context. You need the full triplet.

The most common blind spot involves treating RAG testing like any other LLM assessment. Teams measure the final reply’s quality, but never capture what background data the generator actually worked with. Without that information, half the metrics in this article are impossible to compute.

Next, you should ground your RAG dataset in real human judgment. In our financial assistant, we had a domain specialist on the team who manually QA’d the application from the start. They ran queries, checked whether the right data was retrieved, and verified that the replies made sense.

We translated all of that manual work into our AI evals collection using the error analysis workflow from Article 2.

Also, building RAG datasets introduces a unique difficulty. Each test case needs the right documents, chunks, and embeddings available in the database. Otherwise, the search tool has nothing to work with.

Running the full ingestion pipeline for every evaluation run is slow and introduces variability.

We solved this by coupling each data point with a Postgres SQL export containing the relevant documents, chunks, embeddings, and metadata. We loaded this file directly into the storage system for each test, effectively creating a context cache. This made the process fast and reproducible.

We inject the records, run the query, evaluate the trace, reset the environment, and move to the next item. Image 7 illustrates these data preparation paths.

If you do not have enough production data or expert QA samples, you can create synthetic RAG evaluation sets. Use the reverse workflow from Article 3 by starting from your knowledge base. Use an LLM to extract key facts from specific passages.

Then, formulate realistic user questions that can only be answered using that exact text block.

Because the prompt derives directly from the source material, the input, expected retrieval context, and expected reply are perfectly aligned by construction. This gives you a complete ground-truth triplet. Furthermore, this technique is especially powerful for bootstrapping coverage across your entire document corpus.

Include unanswerable queries in your collection. Do not only formulate prompts that the application should resolve correctly. Instead, create scenarios where the context deliberately lacks the information needed, forcing the agent to refuse or say it does not know.

Without these negative examples, your testing suite is one-sided. Your evals will optimize for always attempting a reply, whereas adding them directly exercises the Answerability metric from Tier 3.

Next, if your RAG architecture integrates with external services, the retrieval path is not just a vector database search. Your agent needs to decide which tool to call first. Should it query Postgres, search the CRM, or check the user’s email?

The best retrieval metrics will not help if your model invoked the wrong data source entirely.

In our financial assistant, this was critical. A query about a client meeting should hit the CRM, not the transaction database. Therefore, we added code-based checks for tool selection alongside our RAG metrics.

Another important trick is to run separate graders per RAG dimension. Do not ask one LLM to evaluate context relevance, faithfulness, and answer relevance in a single prompt. Isolated checks with dimension-specific rubrics produce more consistent results than a unified evaluation.

Ultimately, you need to log specific data for every trace using tools such as Opik. Record retrieved chunks to see what the generator had access to. If faithfulness fails, check whether the reply used information that was not provided. Track metadata such as document IDs and scores, because when context relevance fails, you need to know which items ranked highest. This represents the same observability infrastructure from Article 1.

Next Steps

RAG evaluation is not complex. It is just three variables and six relationships. When your RAG system fails, one of these specific links is broken.

Fix that exact issue and ignore the complexity theater.

Start with Tier 1 retrieval checks as daily prerequisites. Add Tier 2 primary indicators weekly. Extend to Tier 3 when specific failure modes demand it.

Ultimately, match your evaluation priorities to your domain’s risk profile.

Next time you see a vendor dashboard with dozens of RAG metrics, map each one back to the six relationships. If an indicator does not clearly measure one of the core links, it is noise. Drop it and focus on what actually diagnoses failures.

Next up is the final piece in the series. We will explore real-world lessons from months of running evals on a production AI companion. We will discuss what worked, what failed, and what the team would do differently.

Also, remember that this article is part of a 7-piece series on AI Evals & Observability. Here’s what’s ahead:

1. Integrating AI Evals Into Your AI App

2. Build an AI Evals Dataset from Scratch

3. Generate Synthetic Datasets for AI Evals

4. How to Design Evaluators

5. How to Evaluate the Evaluator

6. RAG Evaluation: The Only 6 Metrics You Need ← You just finished this one

7. Lessons from 6 Months of Evals on a Production AI Companion

See you next Tuesday.

Paul Iusztin

What’s your opinion? Do you agree, disagree, or is there something I missed?

Leave a comment

Enjoyed the article? The most sincere compliment is to share our work.

Share

Go Deeper

Go from zero to production-grade AI agents with the Agentic AI Engineering self-paced course. Built in partnership with Towards AI.

Across 34 lessons (articles, videos, and a lot of code), you’ll design, build, evaluate, and deploy production-grade AI agents end to end. By the final lesson, you’ll have built a multi-agent system and a capstone project where you apply everything you've learned on your own.

Three portfolio projects and a certificate to showcase in interviews. Plus a Discord community where you have direct access to other industry experts and me.

Rated 4.9/5 ⭐️ by 300+ students — “Every AI Engineer needs a course like this.”

Learn more

Not ready to commit? We also prepared a free 6-day email course to reveal the 6 critical mistakes that silently destroy agentic systems. Get the free email course.

Thanks again to Opik for sponsoring the series and keeping it free!

If you want to monitor, evaluate and optimize your AI workflows and agents:

Try Opik for free

References

1. Liu, J. (2025, May 19). There Are Only 6 RAG Evals. jxnl.co. https://jxnl.co/writing/2025/05/19/there-are-only-6-rag-evals/

2. Grace, M., Hadfield, J., Olivares, R., & De Jonghe, J. (2026, January 09). Demystifying Evals for AI Agents. Anthropic Engineering Blog. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

Images

If not otherwise stated, all images are created by the author.

She’s deconstructing RAG with a production-first mindset, skipping the theoretical demos to focus on building for architectural reliability.

This one is packed. Let’s get into it 👀 ↓

The “Deer in the Headlights” Moment

I’ve navigated many shifts since the early days of the web, from monoliths to cloud-native microservices and SOAP to REST. But the AI wave felt different. I found myself in a “deer in the headlights” moment, completely unsure of what to learn or even where to start. Should I dive into neural network math, focus on model training, or master context engineering (AI moves quickly)?

Eventually, the path became clear when I realized my real value lay in applying AI to complex business problems. In an enterprise context, that led me straight to RAG. It isn’t just about the model, it’s also about the robust system you build around it. It felt like a return to architecture, a concrete problem to solve where using AI could make a profound difference. However, as I started building, I hit a second roadblock...

Why Most RAG Tutorials Didn’t Help Me Learn RAG

Most RAG tutorials optimize for one outcome: getting an answer out of a model as quickly as possible. That’s fine for demos. It’s a poor way to learn how RAG systems behave in production.

I’m not new to building production software. I’ve spent decades shipping and maintaining systems where debuggability, operability, and failure modes matter. What’s new to me here is RAG, not the discipline of building systems that survive contact with reality. While learning RAG, I wanted to internalize the constraints I’d eventually face anyway: inspectability, idempotent ingestion, debuggable retrieval, and controllable generation. That meant resisting framework-managed chains and owning the control flow early, even if it slowed me down.

This post documents how I’m teaching myself RAG by building a production-grade system in deliberate phases, using frameworks as utilities rather than architecture.

That approach was heavily influenced by, and indeed, inspired by Paul Iusztin’s From 100+ AI Tools to 4: My Production Stack, especially this idea:

AI frameworks are good utilities. They should not dictate the architecture or control flow of your system.

That became my guiding principle.

Before we continue, a quick word from the Decoding AI team. ↓

Go Deeper: Your Path to Agentic AI for Production

Most engineers know the theory behind agents, context engineering, and RAG. What they lack is the confidence to architect, evaluate, and deploy these systems in production. The Agentic AI Engineering course, built in partnership with Towards AI, closes that gap across 34 lessons (articles, videos, and a lot of code).

By the end, you will have gone from “I built a demo” to “I shipped a production-grade multi-agent system with evals, observability, and CI/CD.” Three portfolio projects, a certificate to back it up in interviews, and a Discord community with direct access to industry experts and Paul Iusztin.

Rated 4.9/5 ⭐️ by 290+ early students — ”Every AI Engineer needs a course like this” and ”an excellent bridge from experimental LLM projects to real-world AI engineering.”

Start learning today

↓ Now, back to the article.

The Architecture

Before diving into the details, here is the end-to-end architecture of the RAG system. This diagram serves as a reference model, and we’ll walk through each layer and the production considerations that shaped these choices.

Phase 1. Ingestion: Own the Data

What I built: a pipeline that discovers files → loads documents → normalizes text → chunks → embeds → stores everything in Postgres.

From experience building production systems, ingestion pipelines are where complexity quietly accumulates if they lack idempotence, i.e., the ability to safely re-run without ending up in an inconsistent state, such as duplicate data, partial updates, or stale artifacts. The same applies to traceability, i.e., the ability to trace exactly what happened, to which data, and when. I assumed the same risks would apply here.

What I didn’t account for was how the nature of debugging would differ so vastly from what I was used to in the past. It wasn’t just about emitting log and error information at the right places anymore. A bad chunk doesn’t throw an exception, it just hallucinates an answer three steps later.

Single database, many uses

Instead of introducing a separate vector database, I used Postgres + pgvector. Chunks, embeddings, and metadata live together. That decision buys me a lot:

• I can inspect ingestion results with plain SQL

• I can join vectors with relational metadata

• I can reproduce retrieval behavior outside the application

That inspectability matters when you’re still learning, and having less infrastructure to maintain pays off long after.

Frameworks as utilities, not architecture

I use LangChain’s document loaders (TextLoader, PyMuPDFLoader) for format handling. But the control flow is explicit and mine:

for file_info in discover_files(folder_path):

    raw_docs = load_document(file_info.file_path)

    clean_text = normalize_text(raw_docs)

    chunks = chunk_text(clean_text, chunk_size=512)

    embeddings = await embed_chunks(chunks)

    await save_to_postgres(file_info, chunks, embeddings)

Each step is isolated. Each step can be logged, rerun, or replaced independently. When something breaks, I debug my code, not a framework-managed chain. For instance, during my initial tests, I used PyPDFLoader for the document loading phase. When I inspected the chunking, I realised the chunks had incorrect spaces due to kerning (e.g., ”P r e - C h u n k”). This was easy to address just by swapping PyPDFLoader with PyMuPDFLoader, which handled the complex layouts better.

Idempotence and safe re-runs

I mentioned earlier that pipelines break down when they lack idempotence. Here’s how I addressed it.

Every file’s contents are hashed. If the content hash matches what’s already stored, the file is skipped, no wasted compute, no risk. If the content has changed, its old chunks and embeddings are completely removed before the new ones are written. The database never ends up with a mix of old and new states for the same source.

During development, it makes experimentation safe. For instance, I can tweak chunk sizes, swap embedding models, or change preprocessing logic, then re-run the entire pipeline and trust the result. Without this, every experiment would mean manually cleaning up the database first, or worse, not realizing stale data was still there, silently affecting retrieval quality. More importantly, though, in production, it makes the pipeline resilient to failure. If ingestion crashes halfway through, I can simply restart it. Files already processed are skipped, and the rest pick up where they left off. No manual cleanup, no risk of corruption.

Phase 2. Retrieval: Make Failure Visible

Retrieval is where the quality of your results is determined, which makes debugging discipline more important than clever algorithms.

What I built: query preprocessing → embedding → similarity search → optional reranking.

Most LangChain tutorials show you how to build a RAG pipeline as a “chain,” i.e., a single call where the framework retrieves context, sends it to the LLM, and returns the answer. I chose not to do that. Consistent with the architecture philosophy above, retrieval is an explicit phase, and every step in the retrieval pipeline is an explicit function call I control and invoke directly:

async def retrieve(query: str, top_k: int = 5, rerank: bool = False):

    processed_query = preprocess_query(query)

    query_embedding = embed_query(processed_query)

    results = await search_similar_chunks(query_embedding, top_k)

    if rerank:

        results = rerank_results(query, results, top_k)

    return RetrievalResponse(query=query, results=results)

Keeping retrieval explicit makes failures legible. When an answer is wrong, I can tell whether the issue came from:

• query preprocessing

• embedding quality

• recall

• ranking

Because vectors live in Postgres, I can reproduce retrieval behavior with SQL alone.

That inspectability is invaluable when learning.

Retrieval → Generation boundary

This is the boundary where many RAG systems start to blur failure modes. But they are fundamentally different problems.

Retrieval, including reranking, decides what context is allowed to reach the model. It is a search problem. It fails by missing relevant information (poor recall) or burying it in noise (poor precision).

Generation decides what the model does with the provided context. It is a reasoning problem. It fails by misinterpreting the context, hallucinating facts, or ignoring instructions.

Keeping this boundary explicit helps you immediately diagnose which problem you effectively have. If the answer is wrong but the context contains the truth, you fix the prompt. If the context is missing the truth, you fix the search.

Phase 3. Generation: Treat the LLM as an Unreliable Dependency

What I built: context formatting → LLM invocation with retries → response assembly.

LLMs fail in ways traditional dependencies don’t. They are non-deterministic, occasionally unavailable, and can return plausible but wrong outputs. I treated the model as an unreliable dependency from day one, something to isolate, observe, and swap, not something to trust implicitly.

Swappable LLMs via a factory

A simple factory pattern makes experimentation cheap:

def get_llm():

    if provider == “openai”:

        return OpenAIChat(...)

    if provider == “gemini”:

        return GeminiChat(...)

Switching providers requires only configuration changes. Call sites don’t care. This is exactly where frameworks like LangChain shine: as an abstraction layer. They handle the messy API differences between providers so that OpenAIChat and GeminiChat can expose the same interface to your application. Using them here makes swapping models trivial, without letting them dictate your control flow.

Explicit orchestration over chains

Generation is intentionally step-by-step:

async def generate_answer(request):

    retrieval_response = await retrieve(query=request.query, ...)

    context_text = format_docs(retrieval_response)

    messages = get_rag_prompt().format_messages(

        context=context_text,

        question=request.query,

    )

    llm = get_llm()

    ai_message = await _invoke_llm_with_retry(llm, messages)

    return GenerateResponse(answer=ai_message.content, ...)

I avoided using LangChain’s expression language (LCEL) or runnable abstractions to build this flow. While powerful, they can hide what’s happening. Explicit orchestration is easier to debug, instrument, and reason about, especially while learning. This resonated with me even more since I’m used to a hands-on approach where I can write code and truly understand how the logic flows.

Retries are operational, not semantic

LLM calls fail for mundane reasons: transient network issues, provider-side throttling, or brief outages. I treat those as operational failures, not model behavior, and handle them explicitly.

from tenacity import retry, stop_after_attempt, wait_exponential

@retry(

    stop=stop_after_attempt(3),

    wait=wait_exponential(multiplier=1, min=1, max=10),

)

async def _invoke_llm_with_retry(llm, messages):

    return await llm.ainvoke(messages)

Retries don’t make the model correct, they make the system resilient.

Phase 4. Serving: Thin Adapters, Shared Core

What I built: two interfaces over the same RAG core: a REST API and an MCP server.

In many RAG implementations, the retrieval logic is tightly coupled to the web framework (e.g., defined inside a FastAPI route). This makes it hard to test the logic in isolation or reuse it in different contexts (like a CLI or an evaluation script).

Instead, I treated my RAG system as a standalone library. The core function ‘generate_answer’ takes a pure Pydantic object and returns one. It knows nothing about HTTP, headers, or JSON.

This architecture allowed me to treat serving as a thin adapter pattern.

Adapter 1: REST API (FastAPI)

The REST adapter serves traditional software systems that need deterministic access to the retrieval layer. This includes web applications, backend services, internal tooling, evaluation pipelines, and automation jobs. These are environments where the caller decides exactly when and how the capability should be invoked.

The web layer itself does no extra work. It merely deserializes JSON, calls the core, and serializes the result.

@router.post(”“, response_model=GenerateResponse)

async def query(request: GenerateRequest) -> GenerateResponse:

    return await generate_answer(request)

Adapter 2: MCP Server (Capability Interface for Tool-Using LLMs)

Exposing the same core through the Model Context Protocol (MCP) transforms my RAG pipeline from an application-bound feature into a standardized capability.

MCP standardizes how capabilities are exposed to tool-using LLMs, regardless of whether the caller is a chat assistant, a coding copilot, or an autonomous agent.

I’m used to abstraction via careful refactoring, and it didn’t take long to understand that MCP was just another way of achieving this in the context of AI.

MCP-compatible clients such as Claude Desktop, Cowork, or Cursor can connect to the server and invoke the query_rag tool directly. This allows the underlying LLM to ground its responses in private data without requiring custom integrations, plugins, or connector logic.

Direct tool access is useful, but the MCP interface becomes far more valuable as agents adopt skills to carry out knowledge work and other multi-step tasks. For example, a “Market Research Skill” might combine web search, financial data lookup, and document retrieval. By exposing my RAG system as an MCP Tool, it becomes a standardized block that these skills can easily include in their workflows, without needing custom code.

@mcp.tool()

async def query_rag(query: str, top_k: int = 5, rerank: bool = True) -> dict:

    request = GenerateRequest(query=query, top_k=top_k, rerank=rerank)

    response = await generate_answer(request)

    return response.model_dump()

Both interfaces share the same core logic, thus avoiding duplication. Serving is an adapter problem, not a RAG problem.

Data lineage & traceability

Traceability isn’t new. Long before LLMs, production systems relied on lineage and identifiers to make failures debuggable. LLM non-determinism makes that discipline more important, not less.

Debugging RAG systems almost always means reasoning backward, from an answer, to retrieved chunks, to embeddings, and finally to source files.

In practice, this meant persisting identifiers at every step. Retrieved results carry chunk IDs forward. Generation logs include the IDs of the chunks used as context. When an answer looks wrong, I can trace it deterministically back to its source.

Without lineage, every bad answer looks like a model problem. With it, failures become diagnosable and fixable.

Vendor-neutral observability

This isn’t RAG specific. It’s the same observability discipline I’ve applied in other production systems. I deliberately kept it vendor-neutral, following a pattern I’ve used before to keep core logic decoupled from tooling.

Beyond tracing execution paths, tools like Opik let me reason about operational realities: latency per phase, token usage, and cost per request. Being able to see which model was invoked, how many tokens were consumed, and where time was spent turns performance and cost from assumptions into measurable signals.

def track(name: str = None, phase: Phase = None):

    def decorator(func):

        tags = [f”phase:{phase.value}”] if phase else []

        @opik.track(name=name, tags=tags)

        def wrapper(*args, **kwargs):

            return func(*args, **kwargs)

        return wrapper

    return decorator

If I ever switch observability tools, business code doesn’t change.

What I’m Exploring Next

Next steps include:

1. Adding durable workflow orchestration (DBOS or Prefect)

2. Implementing systematic evaluation for retrieval quality and faithfulness

3. Exploring more advanced retrieval patterns

Each will be added deliberately, one constraint at a time.

Closing Thoughts

Moving from keyword search to semantic and multimodal understanding is a massive leap in how we solve problems. While this technology introduces an ambiguity that contrasts with the deterministic systems I’ve built before, the incredible advantages and sheer problem-solving power it offers make the challenge truly exciting.

Building RAG this way slowed me down, deliberately.

What I have now is a system I can inspect, rerun, and reason about when something goes wrong. For me, that’s a better foundation than a faster demo.

I’m still learning RAG. But I’m learning it with the same instincts that shaped the rest of my career: make systems observable, design for failure, and own the control flow before adding abstraction.

Code: https://github.com/CalvHobbes/rag-101

Inspired by: From 100+ AI Tools to 4: My Production Stack by Paul Iusztin

See you next time.

Priya

What’s your opinion? Do you agree, disagree, or is there something I missed?

Leave a comment

Enjoyed the article? The most sincere compliment is to share our work.

Share

Go Deeper

Go from zero to production-grade AI agents with the Agentic AI Engineering self-paced course. Built in partnership with Towards AI.

Across 34 lessons (articles, videos, and a lot of code), you’ll design, build, evaluate, and deploy production-grade AI agents end to end. By the final lesson, you’ll have built a multi-agent system and a capstone project where you apply everything you’ve learned on your own.

Three portfolio projects and a certificate to showcase in interviews. Plus a Discord community where you have direct access to other industry experts and me.

Rated 4.9/5 ⭐️ by 290+ early students — “Every AI Engineer needs a course like this.”

Learn more

Not ready to commit? We also prepared a free 6-day email course to reveal the 6 critical mistakes that silently destroy agentic systems. Get the free email course.

Images

If not otherwise stated, all images are created by the author.

🧐 Everyone says you need AI evals. Few explain how to actually build them and answer questions such as…

How do we avoid creating evals that waste our time and resources? How do we build datasets and design evaluators that matter? How do we adapt them for RAG? ...and most importantly, how do we stop “vibe checking” and leverage evals to actually track and optimize our app?

This 7-article series breaks it all down from first principles:

1. Integrating AI Evals Into Your AI App

2. Build an AI Evals Dataset from Scratch

3. Generate Synthetic Datasets for AI Evals

4. How to Design Evaluators

5. How to Evaluate the Evaluator ← You are here

6. RAG Evaluation: The Only 6 Metrics You Need

7. Lessons from 6 Months of Evals on a Production AI Companion

By the end, you’ll know how to integrate AI evals that actually track and improve the performance of your AI product. No vibe checking required!

Let’s get started.

How to Evaluate the Evaluator

Your evaluators are running. They produce Pass or Fail verdicts on your agent’s outputs. But one open question remains: how do you know if those verdicts are correct?

While building Brown, a writer agent I developed with the Towards AI team for our Agentic AI Engineering course, I set up an LLM judge to verify generated articles. I wanted to check the expected structure, idea flow, and content against a golden dataset. I ran it on a batch of traces, and the scores seemed reasonable. Then I manually compared the traces against the judge’s verdicts, only to realize it was fixating on the wrong things.

It scored 0 when an article used bullet points instead of H3 headers, which was perfectly acceptable for that section. It scored 0 when the agent used a different transition phrase than the few-shot examples, penalizing creativity when we wanted flexibility. Furthermore, it scored 1 when paragraphs did not flow smoothly into each other, completely overlooking a real quality issue we cared about.

We had to iterate on the judge until it reflected what we actually valued. Anthropic reports a similar pattern, seeing eval scores jump from 42% to 95% after fixing grading bugs and ambiguous task specifications [3]. The agent was fine all along, but the evaluator was broken. That experience crystallized something for me: eval metrics you cannot trust are worse than no metrics at all.

Unvalidated evals create false confidence. You see green dashboards, assume quality is fine, and stop looking. You push broken outputs because the numbers said they were good, and you hear about problems from frustrated users instead of your test suite. Worst of all, you cannot tell which evaluations are wrong, as the 10-20% of incorrect signals hide silently and contaminate every decision built on those scores.

Your evaluator is another AI model that makes binary predictions, so it needs a test set, metrics, and mapped failure modes like any other model.

Also, LLM judges are inherently non-deterministic, meaning they hallucinate, carry biases, and drift. Alignment with human evaluators varies widely by task, with some teams achieving high agreement after careful iteration, while others struggle to break 70% on subjective criteria. The gap between your judge and reality could mean hundreds of bad signals across a thousand evaluations, which you will not know without validation [2].

Here is what you will learn to solve this problem:

• Partitioning your labeled data to prevent data leakage.

• Quantifying agreement using standard classification metrics.

• Systematically closing the gap between your judge and domain experts.

• Dealing with the randomness of LLMs.

To start this process, we first need to structure our dataset correctly.

But before digging into the article, a quick word from our sponsor, Opik. ↓

Opik: Automated Agent Optimization Using Your Data (Sponsored)

This AI Evals & Observability series is brought to you by Opik, the LLMOps open-source platform used by Uber, Netflix, Etsy, and more.

We use Opik daily across our courses and AI products. Not just for observability, but now to automatically optimize our agents’ prompts using the same datasets and metrics we already have in the platform.

You are learning how to build diverse synthetic datasets to evaluate your AI app. But once you have those datasets and metrics, why stop at measuring quality? Opik’s agent optimizer closes the loop. It uses your eval dataset to automatically improve your prompts. Here is why we love it:

• Same datasets, zero extra setup — Opik’s optimizer reuses the exact datasets, metrics, and tracing you already have. Quick start guide.

• Six optimization algorithms — Choose from strategies like HRPO (our favorite), which performs root-cause analysis on failures and proposes targeted fixes, or evolutionary optimization to explore diverse prompt structures. See all algorithms.

• No-code Optimization Studio — For quick iterations, run optimization directly from the Optimization Studio UI. Start from your prompt, pick your dataset, choose an algorithm, and watch Opik test prompt variations against your metrics in real time.

Opik is fully open source and integrates with OpenAI, Anthropic, Gemini, and 100+ providers. Start optimizing your agents for free:

Automated agent optimization guide

↓ Now, let’s move back to the article.

Structuring Your Data for Validation

You already have your ground truth. As explained in Article 2 and Article 3 of the series, your domain expert labeled each trace as Pass or Fail with a critique. Those labels are the reference standard your automated judge must match. If the task is highly subjective, consider having multiple people label the same examples to discover the agreement ceiling, but for most teams, the single expert is sufficient.

Now you need to partition that labeled data correctly. Why? Because you cannot build and validate on the same examples, as that is like grading your own exam. You must calculate the error on unseen data only to make sure you are not getting biased results, so split your dataset into three sets: train, dev and test [2].

The train set takes 60% of the data, representing the examples your evaluator learns from. They go into the few-shot prompt, inform the rubric, and set the standard for what Pass and Fail look like. The dev set takes 20% of the data, acting as your iteration sandbox. Run the judge here, check where it disagrees with the expert, adjust the prompt, and repeat to refine the system. Finally, the test set takes the remaining 20% and must be kept locked until you are done iterating. You use it only at the end when the LLM judge is aligned with the expert on the dev set. This gives you an unbiased final score on data that the evaluator has never seen.

The 60/20/20 split is a good starting point, but as your data grows and you don’t want to overload your few-shot-examples (they grow your context window), you can start moving more data to the dev and test splits.

In practice, 100 labeled examples mean 60 powering the prompt, 20 for tuning, and 20 for the final honest check. Aim for at least 100 labeled examples to get stable metrics. Below 50, your numbers become too noisy to act on. Watch out for class imbalance. If 90% of your traces are Pass and only 10% are Fail, you need a way to balance the classes, either by synthetically increasing your negative class or removing samples from your positive class, until a balance is achieved.

With data properly structured, let us quantify how well your judge actually agrees with the expert.

Measuring Alignment With Human Judgment

Your judge outputs Pass or Fail for each trace, which means you are building a binary classifier. You are using LLMs instead of other models, but ultimately, it’s still just a classifier.

Thus, you need to quantify the performance of the LLM Judge against the golden dataset we just split in the previous section. Standard classification metrics give you this visibility.

The confusion matrix shows four possible outcomes. True Positive (TP) means both judge and expert say Pass, agreeing the output is good. True Negative (TN) means both say Fail, agreeing the output is bad. False Positive (FP) means the judge says Pass, but the expert says Fail, letting a bad output through. False Negative (FN) means the judge says Fail, but the expert says Pass, meaning the judge was overly harsh.

Combining TP, TN, FP, and FN yields three fundamental metrics:

1. Accuracy is the overall agreement rate, calculated as (TP + TN) / total. If the judge matches the expert on 170 out of 200 traces, that is 85% accuracy. This is useful when Pass and Fail are roughly balanced, but it is highly misleading when they are not.

2. Precision measures how trustworthy the Pass verdicts are, representing the fraction of judge-approved traces that the expert also labeled Pass. You calculate it as TP / (TP + FP). If the judge approves 50 articles and the expert disagrees on 8, precision is 42 / 50 = 84%, meaning when the judge says the output is good, you can generally believe it.

3. Recall measures how many actual Passes the judge finds out of all the traces the expert labeled Pass. You calculate it as TP / (TP + FN). If 60 articles are genuinely good but the judge only catches 48, recall is 48 / 60 = 80%, meaning the judge finds most quality output but still misses some.

Ultimately, we have the F1 score as an aggregate metric that provides a balanced view as the harmonic mean of precision and recall, calculated as 2 × (Precision × Recall) / (Precision + Recall). Use this when both false positives and false negatives matter equally. The right F1 target depends on the metric. With Brown, we accepted around 60% for subjective metrics like style, but demanded over 90% for objective ones like article structure. As a general rule, aim for an F1 above 0.70.

These metrics seem simple enough. But there is a common trap most teams fall into when their datasets are not balanced.

When High Scores Hide Real Failures

We can best understand this phenomenon by looking at a few examples.

For example, let’s assume Brown generates 80 articles. 70 are correct, and 10 are broken. Your judge labels every single one as Pass. Accuracy sits at 70 / 80 = 87.5%, which looks reasonable, but it never caught a single failure, making it completely useless.

Let us look at another example in more depth. Out of 80 generated articles, 60 are genuinely well-structured, while 20 have real problems like missing sections or disconnected paragraphs. The judge correctly approves 55 of the good ones and wrongly rejects 5. Of the 20 broken articles, it catches only 4 and lets 16 slip through. That gives us TP=55, FN=5, FP=16, TN=4.

Overall accuracy reads (55 + 4) / 80 = 73.75%, which looks reasonable. But Fail-class recall is just TN / (FP + TN) = 4 / (16 + 4) = 20%, meaning the judge misses 80% of structural failures. The lesson here is to always check precision and recall on the minority class. If those numbers are low, enrich your few-shot prompts with more failure examples, focusing particularly on the subtle cases where individual paragraphs look fine but do not connect fluidly [2].

Now that you know what to measure and what to watch out for, let us walk through the process of systematically improving your judge.

Closing the Gap Between Judge and Expert

This is the core workflow for making your judge reliable. Start with 10-20 few-shot examples from the train set to build your initial judge, and run it against the dev set while leaving the test set untouched. Compute precision, recall, and F1, then identify every disagreement where the judge and expert diverge. Expand your few-shot examples by incorporating those disagreements into the prompt when they reveal real patterns, re-run, and re-measure until the dev set alignment hits your target threshold.

Remember that your few-shot examples translate to input tokens, which translate to extra costs. Thus, ideally, you want to keep your few-shot examples as minimal, yet diverse, as possible, while maximizing performance on your dev and test splits.

Lock the test set for the final check. Only run the judge on the test set after you stop iterating on the dev set, giving you an uncontaminated measurement of real performance.

Expect at least 3 rounds of iteration. If you are still far below target after 10 iterations, the task may require human judgment that no prompt can replicate. Start by hand, as manual prompt refinement teaches you where your judge’s reasoning diverges from the expert’s. Carefully studying each disagreement is the most informative signal you have, and once your labeled dataset is large and high-quality enough, you can explore automated prompt optimization tools.

Read the LLM Judge critiques instead of just looking at metrics, as critiques tell you whether the judge was wrong or the expert missed something. As highlighted by Anthropic, you shouldn’t take eval scores at face value until someone digs into the details and reads the critiques of the judge [3].

Once your judge passes validation, put it to work for regression testing, optimization, and production monitoring as explained in Article 1.

What if the agreement stays low? If after 10 rounds your agreement is still low, here is what to look out for. Your few-shot examples might be too narrow, so as you keep sampling more production traces using your observability platform (e.g., Opik) revisit error analysis, as exlained in Article 2, to find the specific patterns where the judge fails and add those to the few-shot-examples. With Brown, our initial examples were too uniform, and adding subtle structural failures immediately improved alignment.

The rubric might lack specificity, as asking if the article is well-written invites interpretation, while asking if it contains well-defined paragraphs, transitions, and metaphors leaves less room for ambiguity. Sharpen the criteria.

Also, in case the task itself is too subjective, consider accepting a lower F1 score. For example, with Brown, style adherence was inherently subjective, so we accepted a lower F1 there while holding structure to >90%. The idea is to adapt your acceptance threshold based on the nature of each business metric you are tracking.

Even with strong agreement, there is one more challenge. Both your judge and your agent introduce randomness into every run. Let’s see how we can fix that.

Dealing With Non-Determinism

Randomness comes from two directions: as the judge produces different scores on the same input, and the agent itself takes different paths each run. You need to address both to build a stable evaluation pipeline.

The easiest and most powerful way to win is to scale the dataset, as larger datasets smooth out noise. Aim for enough examples in each class that a few misclassifications do not swing your metrics wildly. A good starting point is a minimum of 50 samples per class.

Also, another easy win (but not necessarily cheap) is to pick the strongest available model, using a frontier model like the latest versions of Claude Opus or Gemini Pro, because the judge should be at least as capable as the system it evaluates [2]. Require reasoning before the verdict by structuring the prompt with Chain of Thought (CoT) so the judge walks through each criterion first before delivering Pass or Fail. This step-by-step analysis produces more consistent scores and better alignment with human judgment [1].

Let the judge abstain by giving it an “Unknown” option when it lacks enough information to decide, because forcing a binary Pass/Fail on ambiguous cases generates false positives you cannot distinguish from real ones [3].

To further stabilize the judge, you can compute a significance threshold by running the evaluation 3-5 times and computing the variance between the runs. With Brown, this was essential because writing is subjective, and running the evaluator 5 times told us the real error threshold. A 3% metric shift across runs was noise, but 10% meant something actually changed. Without this, you are chasing random fluctuations.

On the agent side, treat it as a black box and evaluate the destination, not the route, as agents can reach the same outcome through different strategies. Brown might outline first, then write or draft everything, then restructure, but both can produce a strong article. Score the final output against your quality criteria, not the intermediate steps [3].

For the agent, measure reliability across multiple runs using pass@k and pass^k. pass@k tracks whether at least one out of k attempts succeeds, while pass^k tracks whether all k attempts succeed. These two metrics tell opposite stories as k grows: pass@k climbs toward 100% while pass^k dropping sharply, revealing how consistent your agent really is. [3].

You now have the complete toolkit for evaluating your evaluator.

Demo

To fully grasp the end-to-end workflow for building AI Evals, I recommend rewatching our demo using AlignEval, an open-source tool created by Eugene Yan. It provides a streamlined interface for the exact workflow this article teaches: look at your data, label it, evaluate outputs, and optimize your evaluators:

The tool is open source and available at aligneval.com, with the source code on GitHub (eugeneyan/align-app). You can try it for free with your own data or use the prompt below to quickly generate a CSV similar to the one from the demo:

I want you to generate a CSV file with the following characteristics:
"""
* The CSV file must include the following columns:
   * id: Unique identifier for each row
   * input: Context used to generate output
   * output: Generated text to be evaluated
   * label: Ground truth (values optional but counts towards XP)
   * explanation: A one-sentence explanation on why we labeled the row as 0 (PASS) or 1 (FAIL)
* 🚨 The label column only accepts binary labels, either 0 or 1.
   * 0: Output PASSES your evaluation
   * 1: Output FAILS your evaluation
"""
that contains 100 rows

The goal of the CSV file is to implement a dataset to build an LLM Judge evaluator. 

We want to create some mock, synthetic data to conceptually show how labeling, evaluating and optimizing the LLM judge would look like, based on this tool: https://aligneval.com/

Let's say that we collected data from a vertical assistant agent specialized in answering work emails and Slack messages. Thus, create 100 scenarios based on these dimensions:
* feature: email/slack
* scenario: executive, manager, colleague, spam email, phishing email, friend (as an exception)
* label: success/failure of properly answering the message

Where the input is a single email or Slack message or an email or Slack thread, but the output will ALWAYS be just the generated reply, whether it's email or Slack.

Make the labels a 50%/50% split between passes and fails.

Also, note that NO REPLY is an expected behavior for SPAM and phishing emails. Also, for non-essential emails or toxic or slack messages.

We used Claude Opus 4.6 within the Claude app to generate it.

Next Steps

An evaluator only earns trust when it matches expert judgment. The workflow is straightforward: measure where your judge disagrees with the expert, fix those gaps, and confirm on data the judge has never seen. Only when the judge aligns with the expert on the test set can you rely on your eval metrics.

The error analysis workflow and iterative labeling were only the tip of the iceberg. Now you see the full picture of how to build, validate, and maintain evaluators.

Next up is a specialized article focused on evaluating Retrieval-Augmented Generation (RAG) systems.

Also, remember that this article is part of a 7-piece series on AI Evals & Observability. Here’s what’s ahead:

1. Integrating AI Evals Into Your AI App

2. Build an AI Evals Dataset from Scratch

3. Generate Synthetic Datasets for AI Evals

4. How to Design Evaluators

5. How to Evaluate the Evaluator ← You just finished this one

6. RAG Evaluation: The Only 6 Metrics You Need

7. Lessons from 6 Months of Evals on a Production AI Companion

See you next Tuesday.

Paul Iusztin

What’s your opinion? Do you agree, disagree, or is there something I missed?

Leave a comment

Enjoyed the article? The most sincere compliment is to share our work.

Share

Go Deeper

Go from zero to production-grade AI agents with the Agentic AI Engineering self-paced course. Built in partnership with Towards AI.

Across 34 lessons (articles, videos, and a lot of code), you’ll design, build, evaluate, and deploy production-grade AI agents end to end. By the final lesson, you’ll have built a multi-agent system and a capstone project where you apply everything you've learned on your own.

Three portfolio projects and a certificate to showcase in interviews. Plus a Discord community where you have direct access to other industry experts and me.

Rated 4.9/5 ⭐️ by 290+ early students — “Every AI Engineer needs a course like this.”

Learn more

Not ready to commit? We also prepared a free 6-day email course to reveal the 6 critical mistakes that silently destroy agentic systems. Get the free email course.

Thanks again to Opik for sponsoring the series and keeping it free!

If you want to monitor, evaluate and optimize your AI workflows and agents:

Try Opik for free

References

1. Arize AI. (n.d.). LLM as a Judge: Primer and Pre-Built Evaluators. Arize. https://arize.com/llm-as-a-judge/

2. Husain, H. (n.d.). Using LLM-as-a-Judge for Evaluation. hamel.dev. https://hamel.dev/blog/posts/llm-judge/

3. Anthropic. (n.d.). Demystifying Evals for AI Agents. Anthropic Engineering Blog. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

Images

If not otherwise stated, all images are created by the author.

He’s skipping the textbook definitions today to focus on the architectural trade-offs and real-world logic behind his most recent builds.

Enough chitchat. Let’s get into it 👀 ↓

When Single-Agent Systems Fall Apart

You know the moment. You built a perfectly capable AI agent that writes code, answers questions, and searches through your docs. It works great. Then you ask it to review code for security issues and synthesize three different research papers. It returns something that’s half right and half wrong, delivered with full confidence.

I used to think this was a model problem. Better prompts, bigger context window, maybe switch to the latest Sonnet release. Wrong. The problem is architectural, and no amount of prompt engineering fixes it.

A single agent with 40+ tools, a 2,000-word prompt over five different domains, and retrieval tuned for one job at a time collapses. Context windows get bloated. Tool selection becomes a mess. Quality tanks.

This happened to me with Screech, a personal agent I built for my side projects. It started simply, basically a smarter search over my notes. Then I kept adding: code generation, documentation, code reviews, security audits, and research synthesis. The single-agent approach worked beautifully until it very suddenly didn’t.

The stack is not exotic. It’s VoltAgent for runtime and workflows, SurrealDB as the “one DB to store everything” experiment, and Claude as the default model tier.

And yes, the agent is named after Screech from the Saved by the Bell TV series. Also, my childhood nickname.

I didn’t invent this in a vacuum. Graphiti shaped how I think about knowledge that changes over time. VoltAgent gave me workflow primitives I didn’t have to build. Paul Iusztin’s AI Agents Foundations convinced me to stop forcing PDFs through OCR and treat them as images. Agentic Playbooks showed me that auditable agent decisions are a performance win, not only a governance check.

So, here’s the architecture, decisions, and stuff I’d do differently. For legal reasons, that’s “informational only,” not “you should do this.”

Before we continue, a quick word from the Decoding AI team. ↓

Go Deeper: Your Path to Agentic AI for Production

The Agentic AI Engineering course, built in partnership with Towards AI, walks you through building exactly this kind of multi-agent architecture across 34 lessons.

Across 34 lessons (articles, videos, and a lot of code), you’ll design, build, evaluate, and deploy production-grade AI agents end to end. By the final lesson, you’ll have built a multi-agent system and a capstone project where you apply everything you’ve learned on your own.

Three portfolio projects and a certificate to showcase in interviews. Plus a Discord community where you have direct access to other industry experts and me.

Rated 4.9/5 ⭐️ by 290+ early students — ”Every AI Engineer needs a course like this” and ”an excellent bridge from experimental LLM projects to real-world AI engineering.”

Start learning today

↓ Now, back to the article.

If You’re Not Building Agents

You can stop here and still get the gist. The problem: one AI agent that should do everything (search your notes, write code, review for security, summarize research) does all of it poorly. One large prompt and many tools. It confuses tasks, wastes tokens, and returns confident nonsense when goals conflict, e.g., security paranoia versus “ship it” code gen.

The solution: one conductor agent that handles simple work itself and a pool of specialists it calls when the task needs depth. The conductor stays cheap and fast for most requests. Specialists run only when needed. You need routing (who handles what), hybrid retrieval (not only vector search), and one store for documents, relationships, and chat (here, SurrealDB). The rest of this article is for people who want to see the wiring.

Multi-Agent: When It’s Worth the Complexity

The maintenance overhead is real. So let me be clear about when this makes sense.

I’d only do it when there are 3+ domains that actively conflict (dev, research, security is a classic triangle), when I care about cost per request (not “cost later”, cost now), and when I need failures to be contained so that one specialist can be dumb without contaminating the whole system.

Stay single-agent when tasks are similar, the tool count is under about 15, you do not need different model tiers, and simplicity beats per-task quality.

Single-agent favors simplicity. Multi-agent favors quality per task and adds orchestration. Pick your poison.

The Three-Layer Architecture

I think of Screech as an orchestra. One conductor who knows the entire score but doesn’t play every instrument. Backed by specialists who are genuinely brilliant at their specific parts.

Layer 1: Orchestration. It does the boring-but-hard parts: understanding intent, pulling context, and deciding whether this is “handle it now” or “call a specialist”. Three meta-tools carry most of the orchestration weight: discover subagents, invoke one subagent, and fan out to multiple subagents. A task router (Claude Haiku) classifies complexity before any expensive model runs. The runtime, memory management, workflow engine (with suspend/resume), and MCP server integration come from VoltAgent. I didn’t build any of that infrastructure, I plugged in.

Layer 2: Specialists.128 subagents in 10 categories: core development, language specialists (TypeScript, Python, Rust, plus 19 more), testing & quality, meta-orchestration. More on why that number and why these categories in a bit.

Layer 3: Knowledge. Hybrid retrieval combining vector search + knowledge graph traversal + keyword matching, all backed by SurrealDB. Plus a temporal layer (Graphiti-style) so the system knows when it learned something, not only what.

Here’s the key decision: Screech is a full agent with its own tools and retrieval, not a dumb router. That is the decision that matters. It handles 60–70% of requests directly. Subagents only kick in when you need deep specialization. That keeps latency and cost sane for the common case.

If you’ve used Claude Code, this pattern will feel familiar, but there is a key difference: Claude Code is one agent plus injected context (CLAUDE.md, conventions, slash commands). When you give it a task, the same agent handles everything, and it just gets extra context injected from your skill files. It’s the “enhanced single-agent” end of the spectrum: one brain, augmented with domain knowledge. Screech pushes further along that spectrum. Instead of injecting domain knowledge into one agent’s prompt, each specialist is its own agent with a dedicated system prompt, model tier, and tool set. The orchestrator doesn’t just get “React knowledge” injected — it delegates to an react-specialist agent that was born and bred to think in components, hooks, and JSX. The difference matters when domains actively conflict: a security auditor’s “assume everything is dangerous” mindset would poison a code generator’s “keep it simple” prompt if they shared the same context. Separate agents, separate prompts, no cross-contamination. Think of it as: Claude Code = one chef who reads different recipe books depending on the dish. Screech = a head chef who delegates to a pastry specialist, a sushi chef, and a grill master: each with their own kitchen and knives.

The diagram below shows how these layers connect. Here’s the flow from top to bottom:

User Interfaces layer exposes three entry points: a web UI (React), an MCP server (for IDE integration like Cursor), and a CLI terminal. All hit the same orchestration layer.

Screech Web UI is a React-based interface for the Screech personal knowledge agent. It connects to the Screech backend and provides five main views: Sources (ingest and manage documents, view chunk/entity/pattern/insight stats), Notes, Chat (conversation with the agent), Search (query over your knowledge), and Knowledge Graph (browse entities and relationships). It also shows connection status and supports running synthesis. Its main use is to browse, chat, search, and explore your knowledge graph in one place.

Orchestration Layer has as a master agent the Screech agent (Claude Sonnet 4), which sits at the center, handling 60–70% of requests directly. Three supporting components surround it: the Task Router (Haiku, $0.0025/classification), the Event Bus (in-process pub/sub), and Persistent Memory (conversation history, user context).

The 128 Subagents are arranged by category: Core Dev (11), Language Specialists (22), DevOps (15), Testing & Quality (13), Domain-Specific (27), Business (12), Research (6), Dev Experience (12), and Meta-Orchestration (10). The orchestrator delegates to these when deep specialization is needed.

Hybrid Retrieval sits between the agents and the database: 0.6 vector + 0.2 graph + 0.2 keyword, merging three signals before final relevance scoring.

SurrealDB acts as the persistence layer, split into three logical stores in one database: the Vector Store (MTREE index, 3072-dim embeddings, cosine similarity), the Knowledge Graph (entities, relationships, BFS traversal), and the Temporal Graph (Graphiti-inspired episodes, facts, time-range queries).

At a code level, Screech is just an Agent instance with four things wired in: a model, a hybrid retriever, a tool set, and persistent memory. This “agent factory” is the single place where the orchestration decisions become concrete.

// The Screech agent factory
const agent = new Agent({
  name: "Screech",
  purpose: "Unified personal agent for side projects, knowledge synthesis, " +
           "development, documentation, and orchestration of specialist subagents.",
  model: anthropic("claude-sonnet-4-20250514"),
  retriever,  // Hybrid RAG (vector + graph + keyword)
  tools: screechTools,  // Deduplicated from 3 domains
  memory,  // LibSQL persistent memory
});

11 Tables, One Database: The SurrealDB Model

One of the strongest arguments for SurrealDB: documents, embeddings, knowledge graph, temporal events, and conversation memory in 11 tables, one database. No Postgres + Neo4j + Redis dance.

Documents and Chunks

Ingest a document. Create a document record (metadata, content hash for dedup). Then split it into chunk records. Each chunk gets a 3072-dim embedding (OpenAI text-embedding-3-large). SurrealDB’s MTREE index does cosine similarity. MTREE is a tree index for high-dimensional vectors (same idea as pgvector’s HNSW/IVFFlat). It lets SurrealDB find the nearest embeddings without brute-force scanning every row. Chunks are multimodal. They store image_data (base64) and mime_type alongside text. This comes straight from Paul Iusztin’s insight: stop forcing PDFs through OCR. Treat them as images.

// Multimodal chunk structure
interface Chunk {
  document_id: string;  // Parent document link
  content: string;      // Text or image description
  embedding: number[];  // 3072-dim vector (MTREE indexed)
  mime_type?: string;   // "text/plain", "image/png", "application/pdf"
  image_data?: string;  // Base64 for vision-processed pages
  page_number?: number; // PDF page tracking
}

Entities and the Graph (Your Ontology)

Here’s where Screech diverges from typical RAG: I extract a structured knowledge graph. Claude identifies entities and relationships from each document. SurrealDB’s RELATION type makes this straightforward: entity table relates_to with TYPE RELATION IN entity OUT entity. No separate graph DB.

-- SurrealDB graph relationships (native support)
DEFINE TABLE relates_to SCHEMAFULL TYPE RELATION IN entity OUT entity;

RELATE entity:react->relates_to->entity:nextjs CONTENT {
  relation_type: "EXTENDS",
  confidence: 0.9,
  description: "Next.js extends React with SSR and routing"
};

I picked 11 entity types on purpose. This is the system’s ontology, the vocabulary it uses to classify everything it learns: concept, person, organization, tool, technology, pattern, best_practice, principle, process, document, topic. Each type has its own extraction prompt (e.g., person for roles and affiliations, technology for use cases, and ecosystem). Relationship types include IMPLEMENTS, USES, DEPENDS_ON, PART_OF, EXTENDS, SIMILAR_TO. The ontology is deliberately small; there is enough granularity for useful graph queries without turning into a taxonomy nightmare. Bigger ontologies mean more edge cases and more “is this a tool or a technology?” ambiguity. Eleven types cover 95%+ of what a personal knowledge agent encounters.

Episodes and Facts: Temporal Layer (It’s a Log)

This is the Graphiti-inspired layer that most RAG systems completely skip. Every ingestion creates an episode. Think of episodes as an append-only log of everything the system has ever learned. It is time-stamped and immutable. Episodes link to entities via source_episode_ids. Ingest a PDF, and you get a new episode. Process a paper, and you get a new episode. They do not get updated or overwritten. Old episodes don’t get deleted when new ones arrive; they stay in the timeline with their original timestamps. You can ask “what did I know about X six months ago?” and get a real and accurate answer.

Facts are triples (subject, predicate, object) with a source_episode_id. They capture the structured knowledge extracted alongside each episode. When two facts conflict, e.g., “Bun is experimental” (June) and “Bun is production-ready” (January), the agent can prefer the more recent one.

Why does this matter? Knowledge changes. Without temporal tracking, both facts coexist in your knowledge base with equal weight, and the agent might confidently cite the stale one. Graphiti calls this “bi-temporal awareness”. Tracking both when a fact was true in the world and when the system learned it.

Behind the scenes, temporal queries run entity search (match query terms), then episode retrieval in a time range (filter out chat episodes, keep knowledge episodes), then relevance filtering and linking back to entities. The result is a time-ordered context. The context field returned is prompt-ready: entity descriptions and relationship sentences in plain language.

// Time-aware queries
const recent = await queryTemporalGraph("recent learnings", {
  includeTemporal: true,
  timeRange: { start: oneMonthAgo, end: now },
});
// Returns: episodes + linked entities + relationships, time-ordered

Patterns and Insights: The “So What?” Chain

Beyond storage, Screech runs a synthesis pipeline. Detects patterns across your knowledge base. Generates actionable insights. The pattern table stores detected patterns (workflows, successes, failures, optimizations) with embeddings for searchability. The insight table stores generated insights, each linked back to source patterns with relevance scores.

Documents become chunks, which get embedded. Chunks become entities through extraction. Entities become patterns once the synthesis pipeline starts noticing recurring signals. Patterns become insights (actionable takeaways with provenance). Each stage feeds the next. Every layer is searchable. Vague question goes to vector over chunks. Relationships go to a graph over entities. “What should I do?” goes to insights.

Conversation Memory

user, thread, message tables handle conversation memory. Zep-style user summaries, conversation threading, and message history. Persistent context across sessions, but separate from the knowledge base.

The diagram below shows how data flows through the system. Think of it as five layers stacked on top of each other, each feeding the next:

1. Document layer (top): document and chunk. Raw material comes in here. A PDF becomes a document record; its content gets split into chunk records, each with a 3072-dim embedding. This is the foundation everything else builds on.

2. Temporal layer: episode and community. Every ingestion event creates an episode timestamped to when the system learned it. Episodes link back to chunks (what was ingested) and forward to entities (what was extracted). This is the Graphiti-inspired time dimension—the system knows when it learned something, not just what.

3. Knowledge graph layer: entity, relates_to, and fact. Entities extracted from chunks (concepts, technologies, people) live here, connected by typed relates_to edges. The diamond shape in the diagram represents the relationship table, which is a SurrealDB RELATION type that sits between entity nodes. fact triples (subject, predicate, object) capture the structured knowledge extracted alongside entities.

4. Synthesis layer: pattern and insight. Patterns detected across your knowledge base (recurring workflows, success/failure signals, optimization opportunities) and actionable insights generated from those patterns. Each links back to the entities and episodes that sourced it.

5. Conversation layer (bottom): user, thread, message. Conversation memory, separate from knowledge. Threads reference the user; messages reference threads. The agent can query conversation history independently of the knowledge base.

The arrows in the diagram show the key relationships: chunks link to their parent document. Episodes link to chunks and entities. Entities connect via relates_to. Patterns and insights link back to entities and episodes for provenance. Each layer is independently searchable via the hybrid retrieval pipeline.

When to Use This vs. Alternatives

Unified SurrealDB works when you want graph + vector + relational without three databases, your dataset is moderate (thousands to tens of thousands of docs), and you value dev velocity over ecosystem maturity.

For production SLAs, PostgreSQL plus pgvector is the safer bet. If your graph is only 2–3 hops (like Screech’s BFS), Postgres handles it with recursive CTEs or JOINs, even at millions of rows. Neo4j earns its place when you need deep traversals or heavy graph queries. Graphiti uses Neo4j for that. For my 2-hop, few-thousand-entity case, Postgres + pgvector could do it all. I chose SurrealDB to prototype faster with one schema and one connection. Right call for me. One schema file. One connection. One query language. I would not blindly recommend it for a team with compliance needs.

Subagent System: Factory, Registry, Profiles

Every subagent is a factory (memory?) => Agent. Keeps instantiation lazy (no subagent created until needed) and shared memory (agents in the same workflow see the same conversation history).

// Subagent factory pattern
export type SubagentFactory = (memory?: Memory) => Agent;

// Registry metadata for discovery
export interface SubagentDefinition {
  name: string;
  description: string;
  category: SubagentCategory;  // 10 categories
  tags: string[];
  modelTier?: ModelTier;       // fast | standard | reasoning
  toolProfile?: ToolProfile;   // core | dev | security | full
  capabilities?: SubagentCapabilities;
  factory: SubagentFactory;
}

Three decisions that actually matter:

Model tiers control cost. Not every agent needs Sonnet. Simple formatting → Haiku (~90% cheaper). Security audits → reasoning tier. Default is standard (Sonnet 4). Router can override.

const MODEL_MAP = {
  fast: anthropic("claude-3-5-haiku-20241022"),     // ~10x cheaper
  standard: anthropic("claude-sonnet-4-20250514"),   // Balanced
  reasoning: anthropic("claude-sonnet-4-20250514"),  // Same model, deeper prompts
};

Tool profiles prevent token waste. Research analyst doesn’t need git tools. Code reviewer doesn’t need security scanning tools. Four profiles: core (7 tools): Knowledge/RAG + file ops + workflow discovery, dev (15 tools): core + git + code analysis + testing, security (17 tools): dev + security scanning + dependency audit andfull (18 tools): everything (backwards-compatible default)

Each subagent can add domain tools on top of its profile.

Capability declarations. With 128 agents, “find agents tagged typescript” returns a dozen. The orchestrator needs to know what each agent is good at. Each subagent declares what it can do, expected input, output, and latency tier. Semantic matching sends “TypeScript conditional types” to the agent whose canDo includes “conditional types” and “type system design”, not any agent with TypeScript in the tag. Same language field, different canDo, e.g., typescript-pro vs. react-specialist.

capabilities: {
  canDo: ["type system design", "generics", "conditional types"],
  languages: ["typescript", "javascript"],
  inputSchema: "code snippet or type problem description",
  outputSchema: "typed solution with explanation",
  latencyTier: "medium",
}

Why 10 categories? Flat list worked until about 40 agents. Then discovery got noisy. Categories are a coarse filter: the orchestrator picks a category, then finds the right specialist within it. The split follows prompt conflicts: security paranoia vs. code-gen creativity, test-engineer adversarial vs. technical-writer explanatory. Separate categories, separate prompts.

At the same time, agents in the same category share a tool profile but differ in expertise. All language specialists get the dev tool profile (git, testing, code analysis). All testing-quality agents share dev tools, too, but their prompts are tuned for finding problems, not writing code. Security agents get security tools. Research agents only need core tools (knowledge/RAG). The table below summarizes counts and examples. The diagram repeats it visually.

How Voltagent Fits Into the Picture

VoltAgent is the runtime layer that makes the orchestration practical: it provides workflow primitives, tool execution, memory management, and suspend/resume so the orchestrator and subagents can run as a coordinated system.

The $0.0025 Routing Layer

This is what makes the economics work. Before anything expensive runs, I do a tiny classification call (Haiku) to label the request. Complexity. Domain. Suggested tier. Up to three candidate specialists. The system prompt gives Haiku the full category list. Rules are explicit. “Security” or “vulnerability” always goes to reasoning. Simple question goes too fast. Code gen goes to at least standard.

const classificationSchema = z.object({
  complexity: z.enum(["simple", "moderate", "complex"]),
  domain: z.enum(["lookup", "formatting", "code-generation",
                  "code-review", "architecture", "security",
                  "debugging", "research", "orchestration", "other"]),
  reasoning: z.string(),
  suggestedTier: z.enum(["fast", "standard", "reasoning"]),
  suggestedSubagents: z.array(z.string()).max(3),
});

Step 2: Domain overrides. resolveRoutedTier() takes the complexity-based tier and domain overrides and picks the higher of the two. So a “simple” security question still goes to reasoning. Security that looks simple often is not. The override is a safety net for Haiku’s optimism.

const DOMAIN_TIER_OVERRIDES = {
  security: "reasoning",     // No shortcuts
  architecture: "reasoning", // No shortcuts
  debugging: "reasoning",    // No shortcuts
  "code-review": "standard", // At least standard
};

Step 3: Final routing. Resolved tier + suggested subagents + rationale → orchestrator picks model and specialists.

Fallback. If classification fails (network error, timeout), we default to { complexity: "moderate", tier: "standard" }. Fail to the middle. Not cheapest (might undershoot). Not the most expensive (waste on every failure). Safest with zero information.

Cost: ~$0.0025 per classification (~$0.25/M input tokens on Haiku). Route 1,000 tasks, spend $2.50. If even 30% land on Haiku instead of Sonnet, you save on the order $8-10 per 1,000 tasks. The router pays for itself quickly.

Hybrid Retrieval: Three Signals Beat One

I started with pure vector search. It worked until it didn’t. Three failure modes:

1. Structural queries fell flat. “What tools does the API designer use?” The answer is in the relationship structure. Vector search gave me chunks that mentioned the API designer, not the ones describing its tool config. I needed graph traversal.

2. Exact-match queries got paraphrased away. “What is the error for SQLITE_BUSY?” Embeddings map that into “database locking” neighborhood and miss the chunk with the actual error code. I needed a keyword.

3. Long-document questions needed reasoning, not similarity. “What are the conclusions?” The conclusion section often is not the most similar to the word “conclusions”. The intro restating the thesis can score higher. I needed the model to reason over document structure (e.g., a table of contents), not only similarity.

Instead of trying to make one approach handle everything, I split retrieval into three paths:

The agent picks which tool to call: document_search / get_context → hybrid (vector + graph + keyword, then rerank). search_within_document → same pipeline, one document. answer_from_document_deep → build a section tree from chunks, LLM picks sections, fetch those chunks only. No vectors on path 3.

Why Three Signals Beat One (Paths 1 & 2)

The hybrid pipeline runs three queries in parallel, then merges with configurable weights:

const DEFAULT_WEIGHTS = {
  vectorWeight: 0.6,   // Semantic similarity (primary)
  graphWeight: 0.2,    // Relationship traversal
  keywordWeight: 0.2,  // Exact matching
};

Why these weights? Vector search gets 0.6 because most knowledge base questions are conceptual, like “explain X,” “how does Y work?”; therefore, embeddings handle these well. OpenAI’s text-embedding-3-large (3072 dims) with SurrealDB’s native MTREE index, cosine similarity. The threshold is intentionally low (0.35), but you can tweak that value according to your use case. Similarity scores compress into a narrow range, so 0.35 is more selective than it sounds.

Graph search gets 0.2 because structural queries are the minority but high-value. It works in two stages. First, it finds entities that match the query by name or description, basically a simple text search over the entity table (concepts, technologies, people, organizations). Then it expands outward from those matches using breadth-first search (BFS): for each matched entity, it queries the relates_to edges in SurrealDB to discover neighbors, scoring them at 70% of the parent’s relevance. Configurable traversal depth (default: 2 hops) controls how far the expansion goes. It should be deep enough to find meaningful connections, but shallow enough to avoid pulling in the entire graph.

So: vector search finds a chunk mentioning “React”? Graph search starts at the “React” entity node, walks its edges, and pulls in “hooks”, “server components”, or “Next.js”, without needing those terms in the original query. Need the path between two concepts? A separate BFS finds the shortest connection: React → EXTENDS → JavaScript → USES → V8, each hop following typed relationships (IMPLEMENTS, USES, DEPENDS_ON, EXTENDS, SIMILAR_TO). This is the signal that vector search fundamentally cannot provide, because relationships are structural, not semantic.

Keyword search gets 0.2 because sometimes you just need to find the exact string. Ask a pure vector system “what version of React does project X use?” and it’ll confidently return chunks about React 17, React 18.2, and React 19, all because to an embedding model, they’re all basically “React with a number.” Helpful if you’re writing an essay. Useless if you need the actual version pinned in your package.json. Keyword search is the boring friend who actually reads the label. Full-text matching with term coverage scoring. No AI magic, just string comparison. And for error codes, version numbers, and config keys, that’s exactly what you want.

After merging, results go through reranking, and this is where the quality jump happens. The weighted merge gets you close, but reranking catches cases where a high-scoring vector result is semantically related but does not actually answer the question.

The reranker supports three methods, selectable per query:

• Embedding reranking (fast, cheap): recalculates cosine similarity between the query embedding and each result’s embedding, then blends it 50/50 with the original merge score. This catches results that scored well on the graph or keyword but are semantically distant from the actual query. Fast because it’s just math. You don’t need an LLM call.

• LLM reranking (slower, more accurate): sends the query + top 20 candidate passages to Claude Sonnet 4, which scores each on a 0–1 relevance scale. The LLM understands intent, and it knows that “how do I fix CORS errors?” is asking for a solution, not a definition. Sits behind an LRU cache (128 entries, 5-minute TTL) to avoid redundant calls for similar queries.

• Hybrid reranking (two-pass): embedding reranking first to narrow the candidate set, then LLM reranking on the survivors. Best quality, highest latency.

On top of any reranking method, there’s an optional diversity-aware mode using MMR (Maximal Marginal Relevance). It iteratively selects results that maximize relevance while penalizing similarity to already-selected results, so it prevents returning five chunks from the same paragraph. Plus a source-type preference layer that weights chunks, entities, patterns, and insights differently depending on the query type.

Why Reasoning Beats Similarity for Long Documents (Path 3)

This is the insight that took me the longest to internalize. For a 300-page PDF, when someone asks “what are the conclusions?” the location of the answer is a function of document structure, not content similarity. A chunk from the introduction that restates the thesis will often score higher on cosine similarity to “conclusions” than the actual conclusion section. More embedding dimensions won’t fix this. Better chunking strategies help, but don’t solve it.

Path 3 skips vector search entirely. The pipeline has three steps:

Step 1: Build the section tree. Take the document’s chunks (already stored from ingestion) and group them into sections. If chunks have page numbers (PDFs), group by page. Otherwise, group into fixed-size windows (default: 4 chunks per section). Each section node gets an ID, a title (”Page 12” or “Section 5”), and a short summary (first ~220 characters of the first chunk). The result is a flat list of TreeNode objects, essentially a reconstructed table of contents.

interface TreeNode {
  node_id: string;        // "s0", "s1", "s2"...
  title: string;          // "Page 12" or "Section 5"
  summary: string;        // First ~220 chars of first chunk
  startChunkIndex: number;
  endChunkIndex: number;
  pageRange?: string;     // "pp. 12–14"
}

Step 2: LLM selects relevant sections. The tree outline (node IDs + titles + summaries) is sent to Claude in a single prompt. The key instruction: use reasoning, not keyword matching. The prompt explicitly tells the LLM to think structurally, e.g., “conclusions are usually in the final section”, “see Appendix G means look for an appendix section.” The LLM returns a JSON object with its reasoning and a list of selected node IDs.

// What the LLM sees (abbreviated)
// - s0: Page 1 — "Chapter 1: Introduction. This paper presents..."
// - s1: Page 2 — "Related work in retrieval-augmented generation..."
// - ...
// - s14: Page 28 — "7. Conclusions and Future Work. We have shown..."

// What the LLM returns
{
  "thinking": "Conclusions are in the final sections. s14 title mentions Conclusions.",
  "node_list": ["s14"]
}

Step 3: Fetch and return. Map the selected node IDs back to chunk index ranges, fetch those chunks, and concatenate their content. That’s your retrieval context. No embedding comparison anywhere in the pipeline.

The fallback is important: if the LLM returns invalid JSON or no valid node IDs, the system defaults to the first 2–3 sections. Better to return something than nothing, and introductory sections are a reasonable default for most questions.

The design is directly inspired by PageIndex‘s thesis: similarity ≠ relevance, and reasoning over document structure often beats embedding search for professional long-form content. It won’t help for vague conceptual questions; that’s what path 1 is for. But for “where in this document does X live?” or “what does chapter 7 say about Y?”, it’s dramatically better because the LLM can reason about document organization the way a human reader would: by scanning the table of contents first.

Document-scoped (path 2) simply narrows the same hybrid pipeline to one document via a documentIds filter. Same three signals, same reranker—just scoped.

The temporal knowledge graph is directly inspired by Graphiti, Zep’s framework for building real-time knowledge graphs. Their core insight: knowledge isn’t static. Tracking when facts were learned matters as much as the facts themselves. Perfect for a personal agent that continuously ingests new content.

Every ingested piece creates an “episode”; remember that it’s a timestamped event linking to extracted entities and fact triples (subject-predicate-object). So now you can use time-aware queries: “What technologies have I been reading about this month?” or “How has my understanding of RAG changed?”

Four Production Patterns That Actually Saved Me

These emerged from running this thing in the wild. I’d recommend all four to anyone building multi-agent systems.

1. LLM Resilience with Tiered Timeouts

Every LLM call goes through withLLMResilience():wrapper that adds per-attempt AbortController timeouts, exponential backoff with jitter, retry only for rate limits / 5xx/ network. Never retries 4xx errors (your fault, not theirs). Different timeouts per use case: classification 60s (if it takes that long, fail), synthesis 300s (different budget). I learned this the hard way. One stuck call should not hold everything up.

export const LLM_TIMEOUTS = {
  fast: { timeoutMs: 60_000, maxRetries: 3 },      // Classification, reranking
  standard: { timeoutMs: 120_000, maxRetries: 2 }, // Agent generation
  long: { timeoutMs: 300_000, maxRetries: 1 },     // Synthesis, deep analysis
};

// Usage
const result = await withLLMResilience(
  (signal) => anthropic.messages.create({ ... }, { signal }),
  { ...LLM_TIMEOUTS.fast, label: "task-classification" }
);

Key insight: different operations need different timeout budgets. Classification call taking 60 seconds? Failed. Synthesis operation taking 60 seconds? Just warming up.

2. Findings Cache for Review Chains

When you run multiple reviewers on the same code (code-reviewer → security-auditor → design-analyst), each reviewer produces findings that downstream reviewers need. Without sharing, every reviewer re-parses the same files, re-discovers the same structure, and wastes tokens on duplicate analysis.

The FindingsCache is a singleton in-memory cache keyed by chain ID. It stores two things: structural analysis (file structure, dependencies, symbols, complexity metrics) produced by the first reviewer, and accumulated findings from every reviewer in the chain—each typed with category, severity, source, and location.

// Each finding is typed and traceable
interface ReviewFinding {
  source: string;     // Which reviewer produced it
  category: "structural" | "quality" | "security" | "design" | "performance";
  severity: "info" | "low" | "medium" | "high" | "critical";
  summary: string;    // Human-readable
  data?: Record<string, unknown>;  // Structured data per reviewer type
  location?: string;  // File/line reference
}

The first reviewer in the chain caches the expensive structural work. The next ones get getChainContextForReviewer(): previous findings + structural cache as a prompt-ready string. Typed findings (source, category, severity, location). TTL 10 min, cap 50 chains. Cuts chain latency by about 40%. The expensive part is parsing and context building, not the LLM. Pattern credit: Agentic Playbooks. Traceable decisions are also a performance win.

// First one caches structure
findingsCache.setStructuralCache(chainId, {
  fileStructure: "src/api/users.ts - 245 lines, 3 exports",
  dependencies: ["express", "zod", "prisma"],
  symbols: ["createUser", "validateInput", "UserSchema"],
  metrics: { cyclomaticComplexity: 12, loc: 245 },
});

// Next get pre-built context
const context = findingsCache.getChainContextForReviewer(chainId, "security-auditor");
// Returns previous findings + cached structure, prompt-ready

What the Workflow Looks Like in Practice

The screenshot below is a real run of a multi-step workflow, showing the chain of specialist calls and the event-style logging that makes the system debuggable.

3. In-Process Event Bus

Agents need to share findings without tight coupling. Solution: a singleton in-memory pub/sub: typed events, well-known topics (VULNERABILITY_FOUND, CODE_REVIEW_COMPLETE, etc.), source agent ID + correlation ID for tracing across a review chain, and a typed payload. Security auditor finds a vulnerability? Publishes to vulnerability_found. Code reviewer subscribes, incorporates the finding.

The implementation is a single AgentEventBus class with no external dependencies.

// Typed event structure
interface AgentEvent<T = unknown> {
  id: string;           // Auto-generated: evt_<timestamp>_<counter>
  topic: string;        // Well-known topic (e.g., "vulnerability_found")
  source: string;       // Publishing agent name
  data: T;              // Typed payload
  timestamp: string;    // ISO timestamp
  correlationId?: string; // Chain/session tracing
}

The key design choice: fire-and-forget delivery. When an agent publishes, subscribers are notified via Promise.allSettled(). Slow or failing subscribers never block the publisher. Handler errors are caught and logged, never thrown.

// Publisher (security-auditor)
await eventBus.publish("vulnerability_found", "security-auditor", {
  severity: "critical",
  type: "sql-injection",
  location: "src/api/users.ts:42",
});

// Subscriber (code-reviewer) registered at startup
eventBus.subscribe("vulnerability_found", async (event) => {
  // Incorporate into review findings
});

Late-joining subscribers can replay event history (last 50 per topic). Events auto-expire via TTL (5 minutes) with periodic cleanup every 100 events. This way, it keeps memory bounded without needing a background timer. Source filtering lets subscribers only receive events from specific agents.

4. Live Evaluation with Sampling

Production traffic gets evaluated by moderation and relevancy scorers at configurable sampling rates. Moderation runs on 20% of requests (cheap). Relevance scoring on 10% (LLM judge, expensive). Both async, non-blocking. Never slow down user-facing response.

What I’d Change: Real Talk

The in-memory event bus doesn’t survive restarts. Fine for a personal side-project agent. Terrible for a production system serving a team. Durable workflow engines like Prefect, Temporal, or DBOS solve this really well with less infrastructure overhead than rolling your own durability with Redis Streams or NATS. Current design optimizes for simplicity, not resilience. I made that trade knowingly.

128 subagents is ridiculous. Pareto wins: ~20 agents do 80%+ of the work. The long tail exists because adding a subagent costs basically nothing (factory + registry entry). I should prune the ones that never get used. Future me will regret not doing that earlier.

SurrealDB as a unified store is elegant but young. Graph + vector + relational in one database? Architecturally clean. But doc gaps cost me time. For strict SLAs, I would use Postgres plus pgvector. For 2 to 3 hop graphs, Postgres is enough. Neo4j, when you need a deep graph. I chose SurrealDB to move fast with one DB. Wouldn’t push it on a team with compliance requirements.

Haiku routing adds ~500ms latency. Noticeable in interactive chat. Negligible in background workflows. For latency-critical paths, consider static routing rules (if the tool is security_scan always use reasoning tier) and only invoke the dynamic router for ambiguous tasks.

Workflow suspend/resume is powerful but adds state complexity. The 70+ workflows support human-in-the-loop via suspend/resume—workflow pauses, waits for human input, and continues. Great for approval flows (expense reports, code reviews). Terrible for state management. Every suspended workflow is a piece of state that can go stale. I’ve had workflows suspended for weeks because I forgot about them. Take that as you will.

The Elephant in the Room: Multi-Agent Is Hard for Everyone

Screech and others: Claude Code with Skills, custom orchestrators, the lot.

Overconfidence. One wrong assumption in step 2 of a 10-step workflow and you get a confidently wrong result. Isolated specialists with focused prompts help, but don’t remove it. I still see invented APIs and wrong architectural assumptions.

More agents do not mean better output. Coordination overhead, conflicting findings, more for the human to reconcile. Findings cache and event bus make communication explicit and traceable, but someone still has to review. Synthesis is an LLM summarizing other LLMs. The chain can be long.

Oversight tax. You spend more time reviewing and redirecting than writing. PR review times go up in high-adoption teams (e.g., plus 91%). Comprehension debt: the more you delegate, the less you understand your codebase. Review becomes rubber-stamping. Screech does not fix that.

Token bloat. Tool schemas, prompts, skills. You can blow past 50k tokens before the agent does useful work. I keep tool profiles tight (7–18 per agent). Complex runs still burn tokens.

Credentials. For a side project, it’s manageable. For production with real APIs and DBs, auth and secrets become a project. A lot of agent efforts reportedly fail to scale there. Not the AI. The plumbing. (My therapist has asked me not to elaborate.)

I’m building Screech knowing these limits. The design mitigates some of it. It does not remove it. Multi-agent amplifies both capability and failure modes. Build guardrails.

Standing on Shoulders

VoltAgent. TypeScript agent framework. Runtime, memory, workflows, MCP, observability. Saved me months.

Graphiti (Zep). Temporal knowledge graph. Episodes, bi-temporal awareness. “Knowledge changes” shaped my RAG thinking.

Decoding AI, AI Agents Foundations (Paul Iusztin). Treat docs as images, not OCR. ReAct, tools, memory.

Agentic Playbooks (Justin Narracott). Traceable, auditable decisions as a performance pattern. Findings cache and review chains owe a lot here.

PageIndex (Vectify AI). Reasoning over document structure instead of pure similarity. Path 3 (tree-search) is inspired by this.

Three Patterns Worth Stealing

You don’t need 128 subagents or a temporal knowledge graph. Here are the three ideas that transfer to any multi-agent system:

1. Route cheap before routing expensive. A $0.0025 classification call that routes 30% of tasks to a model, 90% cheaper? Pays for itself on the first batch. Even without subagents, using a small model to decide whether a task needs your large model is almost always worth it.

2. Not every agent needs every tool. Tool profiles cut token usage, improve tool selection accuracy, and make prompts focused. A research analyst with 7 tools outperforms the same analyst drowning in 18 tools they’ll never use.

3. Hybrid retrieval beats any single method. Vector search handles 70% of queries. Graph traversal and keyword matching cover the other 30% (structural queries, exact-match lookups, relationship questions that embeddings silently botch).

The multi-agent pattern isn’t inherently better. It’s a trade: quality per task versus orchestration complexity. Start with a single capable agent. When quality degrades across diverse tasks, reach for these patterns. The hard part isn’t the agents. It’s the routing, the retrieval, the resilience.

Screech runs on VoltAgent (agent framework), SurrealDB (multi-model database), and Anthropic Claude (LLM). Architectural inspiration from Graphiti, Agentic Playbooks, PageIndex, and the Decoding AI community. Built for personal side-project workloads. Adapt the patterns to your scale.

‘Till next time

Lucian Lature | LinkedIn

What’s your opinion? Do you agree, disagree, or is there something I missed?

Leave a comment

Enjoyed the article? The most sincere compliment is to share our work.

Share

Go Deeper

Go from zero to production-grade AI agents with the Agentic AI Engineering self-paced course. Built in partnership with Towards AI.

Across 34 lessons (articles, videos, and a lot of code), you’ll design, build, evaluate, and deploy production-grade AI agents end to end. By the final lesson, you’ll have built a multi-agent system and a capstone project where you apply everything you’ve learned on your own.

Three portfolio projects and a certificate to showcase in interviews. Plus a Discord community where you have direct access to other industry experts and me.

Rated 4.9/5 ⭐️ by 290+ early students — “Every AI Engineer needs a course like this.”

Learn more

Not ready to commit? We also prepared a free 6-day email course to reveal the 6 critical mistakes that silently destroy agentic systems. Get the free email course.

Images

If not otherwise stated, all images are created by the author.

🧐 Everyone says you need AI evals. Few explain how to actually build them and answer questions such as…

How do we avoid creating evals that waste our time and resources? How do we build datasets and design evaluators that matter? How do we adapt them for RAG? ...and most importantly, how do we stop “vibe checking” and leverage evals to actually track and optimize our app?

This 7-article series breaks it all down from first principles:

1. Integrating AI Evals Into Your AI App

2. Build an AI Evals Dataset from Scratch

3. Generate Synthetic Datasets for AI Evals

4. How to Design Evaluators ← You are here

5. How to Evaluate the Evaluator

6. RAG Evaluation: The Only 6 Metrics You Need

7. Lessons from 6 Months of Evals on a Production AI Companion

By the end, you’ll know how to integrate AI evals that actually track and improve the performance of your AI product. No vibe checking required!

Let’s get started.

How to Design Evaluators

You have a dataset. You’ve manually labeled examples. You’ve fixed the obvious bugs. Now you need evaluators that can run automatically and catch problems before users do.

But here’s what trips up most teams: they build evaluators that check for things nobody cares about, or they use off-the-shelf metrics that sound impressive but don’t match their actual use case.

Three months ago, I spent a weekend building what I thought was a comprehensive evaluation suite for an AI agent that drafted replies to customer support tickets. I had ROUGE scores, BLEU scores, semantic similarity metrics, the works. Everything from the NLP textbook.

Then I ran it on production traces. The evaluators gave perfect scores to replies that were factually wrong, missed the customer’s actual question, and used the wrong tone for frustrated users. Meanwhile, they penalized perfectly good replies for using “different words than the reference answer.”

That’s when I realized: generic metrics optimize for academic benchmarks, not business outcomes. (And no, I’m not saying academic metrics are useless. They’re just solving a different problem than “did this agent do what my users needed?”)

The solution is to design evaluators that match your specific success criteria. Not what worked for someone else’s summarization task. Not what scored well on SQuAD. What actually matters for your users in your use case.

In this article, we will cover:

• The evaluation harness: infrastructure that runs evals end-to-end

• Dataset and metric types: direct scoring vs. pairwise vs. reference-based

• Model evaluation vs. app evaluation (and why benchmarks lie)

• Components of an evaluator: reference examples, metrics, rubrics

• When to use code-based checks vs. LLM judges

• Common mistakes (and how to avoid them)

• Advanced metric designs for multi-turn conversations and agentic workflows

Before digging into the article, a quick word from our sponsor, Opik. ↓

Opik: Open-Source LLMOps Platform (Sponsored)

This AI Evals & Observability series is brought to you by Opik, the LLMOps open-source platform used by Uber, Etsy, Netflix, and more.

We use Opik daily across our courses and AI products. Not just for observability, but to design and run the exact evaluators this article teaches: custom LLM judges, code-based checks, and experiments. All from the same platform.

This article shows you how to design evaluators. Opik gives you the harness to run them at scale. Here is how we use it:

• Custom LLM judges with rubrics — Build the evaluators this article describes: define your criteria, add few-shot examples, and run them across hundreds of traces automatically.

• Run experiments, compare results — Test different prompts, models, or configurations side by side. Opik scores each variant with your evaluators and shows you which one wins.

• Plug evaluators into production — The same LLM judges you design for testing run on live traces too. Set up alarms when scores drop below your threshold so you catch regressions before users do.

Opik is fully open-source and works with custom code or most AI frameworks. You can also use the managed version for free (with 25K spans/month on their generous free tier):

Try Opik for free

↓ Now, let’s move back to the article.

Understanding the Evaluation Harness

You can’t manually run 500 test cases. You need automation.

The infrastructure that runs evals end-to-end is called an evaluation harness (1). It loads your dataset, executes your agent on each test case, captures all the outputs and traces, runs your graders, and aggregates the scores into something you can actually use.

Think of it like pytest for AI apps. Except instead of checking if a function returns the right type, you’re checking if an LLM generated text that accomplishes a business goal.

Here’s what a harness does:

1. Loads tasks from your evaluation dataset

2. Provides instructions and tools to the agent (system prompts, available functions, etc.)

3. Runs tasks (often in parallel across multiple trials because LLM outputs vary)

4. Records everything: inputs, outputs, tool calls, reasoning traces, intermediate states

5. Runs graders on the results (your evaluators)

6. Aggregates scores across trials and tasks

Without a harness, you’re manually running your agent on test cases and eyeballing the output. With a harness, you run 500 test cases overnight and wake up to a report showing exactly which failure categories spiked [1].

The harness is separate from your evaluators. The evaluators decide what “good” means. The harness handles the boring work of running everything at scale and collecting results.

Popular harness options include Opik (what we use), Braintrust, LangSmith, and open-source frameworks like Promptfoo. But honestly, you can build a minimal harness in ~100 lines of Python if you need custom logic [1]. The hard part isn’t the infrastructure - it’s assembling the right context (system prompts, conversation history, retrieved docs, tools) for each task. The key is having one. Don’t manually run evals.

Now let’s talk about what those evaluators actually check.

Dataset and Metric Types: Three Ways to Grade

When designing an evaluator, you need to pick a grading strategy. There are three main approaches, each suited for different situations.

1. Direct Scoring (Pointwise Evaluation)

The evaluator looks at a single output and scores it in isolation. No comparison to anything else.

Example:

• Input: “Refund my order #12345”

• Output: “I’ve processed your refund for order #12345. You’ll see the credit in 3-5 business days.”

• Score: Pass (correctly identified the task, provided timeline, professional tone)

When to use:

• You have clear, absolute quality criteria (was it helpful? was it safe? did it call the right tool?)

• You want to track performance over time on the same dataset

• Your baseline is “good enough” not “better than X”

Metrics:

• Binary pass/fail

• 0-1 scores (where 1 = perfect)

• Classification labels (Helpful/Neutral/Harmful)

2. Pairwise Comparison

The evaluator compares two outputs and picks which one is better.

Example:

• Input: “Refund my order #12345”

• Output A: “Refund processed.”

• Output B: “I’ve processed your refund for order #12345. You’ll see the credit in 3-5 business days.”

• Winner: Output B (more informative, sets expectations)

When to use:

• Comparing two model versions (baseline vs. candidate)

• A/B testing different prompts

• LLMs are better at ranking than absolute scoring

Watch out for biases (2):

• Position bias: LLMs favor the first or last response shown

• Verbosity bias: LLMs prefer longer answers even when they’re not better

• Self-enhancement bias: LLMs favor outputs from themselves over other models

You can mitigate these by randomizing response order and using multiple trials.

3. Reference-Based Evaluation

The evaluator compares the output to a known “gold standard” answer.

Example:

• Input: “What’s the capital of France?”

• Output: “Paris”

• Reference: “Paris”

• Score: Exact match (Pass)

Example 2 (Semantic equivalence):

• Input: “Summarize the refund policy”

• Output: “Customers can return items within 30 days for a full refund if unused.”

• Reference: “Full refunds are available for unused products returned within 30 days of purchase.”

• Score: Pass (different wording, same meaning)

When to use:

• You have ground truth answers (FAQs, knowledge bases, structured tasks)

• Task has a single correct answer or small set of acceptable answers

• You’re testing retrieval accuracy or factual correctness

How to measure:

• Exact match: For structured outputs (dates, product IDs, categorical values)

• Semantic similarity / LLM judges: For natural language, where multiple phrasings are valid (summaries, explanations, instructions)

Common metrics (3):

• Exact match

• ROUGE (recall-oriented, good for summarization)

• BLEU (precision-oriented, originally for translation)

• BERTScore (semantic similarity using embeddings)

• LLM judges (for nuanced semantic equivalence)

The trap: Exact match metrics penalize valid variations. If your reference says “The meeting is on Friday” and your agent says “The meeting is scheduled for this Friday,” exact match fails. This is where semantic similarity metrics (BERTScore) or LLM judges become powerful - they can recognize that different phrasings convey the same outcome.

Model Evaluation vs. App Evaluation (Why Benchmarks Lie)

Here’s a distinction that matters more than people realize:

Model evaluation measures the LLM itself, in isolation, on generic tasks. This is what benchmarks like MMLU, HumanEval, and Chatbot Arena do.

App evaluation measures your entire application (LLM + prompts + tools + retrieval + business logic) on your specific use case.

High MMLU score doesn’t mean it handles your refund policy correctly. Benchmarks test general capability. You need to test your specific use case.

Model Evaluation (Benchmarks)

Tests: “Can this LLM answer random trivia, write code snippets, or score high on standardized tests?”

Useful for:

• Comparing foundation models across the board

• Understanding general capabilities

• Academic research

Useless for:

• Predicting whether it will handle your refund policy correctly

• Knowing if it will escalate frustrated customers at the right time

• Determining if it respects your company’s tone of voice

App Evaluation (What You Actually Need)

Tests: “Does my customer support agent correctly process refunds, handle escalations, and follow our policies?”

This is what matters because your users don’t care if GPT-5 scored 95% on MMLU. They care if it solved their problem.

Your evaluators must be grounded in your business use case, not generic academic benchmarks. This means:

• Testing against your actual policies, not Wikipedia facts

• Using your real user queries, not synthetic textbook questions

• Measuring outcomes that impact revenue, retention, or safety

Benchmarks tell you which LLM is “generally smarter.” App evals tell you which version of your system works better for your users.

Don’t mistake one for the other.

Components of an Evaluator

Now that you know the types, let’s build one. Every evaluator has three components:

1. Reference Examples (Few-Shot Prompts)

These are the labeled examples from your dataset. They show the evaluator what “good” and “bad” look like for your specific task.

Remember from Article 2: the real power isn’t in the system prompt, it’s in these few-shot examples. They encode your domain expert’s judgment.

Example:

Example 1 - PASS

Input: “I need a refund for order #12345”
Output: “I’ve processed your refund. You’ll see the credit in 3-5 business days.”
Reason: Confirms action, sets timeline, professional tone.

Example 2 - FAIL

Input: “Can you waive the late fee on my account?”
Output: “I can help with that!”
Reason: Didn’t actually take action or explain next steps. Empty promise.

2. Metrics

The quantifiable measurement of quality. This can be:

• Objective: Did it call the right tool? Is the JSON valid? Is the response under 200 words?

• Subjective: Was it helpful? Was the tone appropriate? Did it follow the conversation flow?

For objective metrics, use code-based checks (fast, cheap, deterministic).

For subjective metrics, use LLM judges or human evaluation.

3. Rubrics

For subjective metrics, you need a rubric: explicit criteria that define what you’re measuring.

Bad rubric:
“Was the response helpful?”

(Too vague. Helpful how? To whom? Compared to what?)

Good rubric:
“Did the response: (1) correctly identify the user’s request, (2) provide a specific action or next step, (3) include a timeline or expectation, and (4) maintain professional tone?”

Rubrics force precision. They make subjective judgments repeatable. These criteria become part of your LLM judge’s system prompt.

Code-Based Evaluators: Fast, Cheap, Objective

Some checks are deterministic. Did the agent call refund_order()? Is the output valid JSON? Does it include a required disclaimer?

Use code for these. It’s faster, cheaper, and never gives you a different answer on the same input.

Use code-based evaluators for:

• Tool calls: Did it call refund_order() with the right parameters?

• Format checks: Is the output valid JSON? Is it under the character limit?

• Required elements: Does it include a disclaimer? Does it have a timestamp?

• Prohibited content: Does it contain banned phrases or leaked data?

Example (pseudocode):

def evaluate_refund_agent(trace):
  # Check if right tool was called
  if “refund_order” not in trace.tool_calls:
  return {”pass”: False, “reason”: “Didn’t call refund_order”}

  # Check if order_id parameter was provided
  params = trace.tool_calls["refund_order"].parameters
  if "order_id" not in params:
    return {"pass": False, "reason": "Missing order_id parameter"}

  # Check if response includes timeline
  if not any(word in trace.output.lower() for word in ["days", "week", "timeline"]):
    return {"pass": False, "reason": "No timeline provided to customer"}

  return {"pass": True, "reason": "All checks passed"}`

Code-based evaluators are:

• Fast: Milliseconds per check

• Cheap: No API costs

• Reproducible: Same input always gives same result

• Easy to debug: When they fail, you know exactly what broke

But they can’t handle nuance. They can’t judge tone, helpfulness, or conversational flow. For that, you need LLM judges.

These code-based evaluators work exactly like classic unit tests you’re already familiar with. They’re deterministic, fast, and easy to debug. That’s why you should always try to implement code-based checks first before reaching for LLM judges. If you can check it with code, do that. Only use LLM judges when code can’t capture what you need to measure.

LLM Judges: Flexible, Scalable, Nuanced

An LLM judge is an LLM that grades another LLM’s output. You give it the task, the output, and the evaluation criteria, and it returns a score with reasoning

LLM judges work in two modes: evaluating outputs against absolute criteria (is it helpful? professional? accurate?) or comparing outputs to reference answers when you have ground truth but need semantic understanding rather than exact string matching.

Use LLM judges for:

• Tone: Was it empathetic? Professional? Not condescending?

• Helpfulness: Did it actually answer the question or deflect?

• Conversation flow: Did it maintain context across turns?

• Reasoning quality: Did the agent’s plan make sense?

How it works:

1. You provide:

   • The input (user query)

   • The output (agent’s response)

   • The context (system prompt, retrieved docs, conversation history)

   • Evaluation criteria (what you’re checking for)

   • Few-shot examples (labeled passes and fails)

2. The LLM judge outputs:

   • A score (pass/fail or 0-1 scale)

   • A critique explaining why

Example prompt (simplified):

You are evaluating customer support responses. For each trace, output Pass or Fail 
with reasoning.

Evaluation criteria:
1. Did the response correctly identify the customer’s request?
2. Did it provide a specific action or next step?
3. Did it include a timeline or expectation?
4. Did it maintain a professional tone?

Here are examples of how a domain expert judged similar cases:

[Few-shot examples from your labeled dataset]

Now evaluate this trace:
Input: [customer query]
Output: [agent response]
Context: [system prompt, policies]

The judge generates:

FAIL

The response correctly identified the refund request (criterion 1: pass) and 
maintained professional tone (criterion 4: pass). However, it didn’t specify a next 
step beyond “we’ll look into this” (criterion 2: fail) and provided no timeline 
(criterion 3: fail). Customer is left waiting with no expectations set.

Strengths of LLM Judges

• Flexible: Handle open-ended tasks where code can’t

• Scalable: Grade thousands of traces automatically

• Explainable: Critiques show reasoning, helping debug failures

Weaknesses of LLM Judges

• Non-deterministic: Same input might get different scores across runs

• Expensive: Every evaluation is an API call

• Needs calibration: Must align with human judgment (we cover this in Article 5)

Making LLM Judges More Stable

1. Use the most capable model (e.g., Claude Opus, GPT-4o) + footnotes (4)

2. Add chain-of-thought reasoning before scoring (”Let’s think step-by-step...”)

3. Control for verbosity bias (normalize response lengths)

4. Run multiple trials and average scores for critical evals

5. Increase dataset size to at least 50-100 samples (reduces noise)

Common Mistakes (And How to Avoid Them)

Mistake 1: Not Providing Critiques

Wrong:
Score: 1

Right:
Score: 1

Critique: “Response correctly identified the refund request but didn’t provide a timeline. Customer left without expectations.”

Critiques are not optional. They’re how you debug failures and train better evaluators.

Mistake 2: Overly Terse Critiques

Wrong:
“Bad tone”

Right:
“Response used dismissive language (’just wait’) when customer expressed frustration about a delayed order. Should have acknowledged frustration and provided specific next steps.”

The critique should be detailed enough to serve as a few-shot example later.

Mistake 3: Missing Context

Don’t evaluate the output in isolation. Give the evaluator everything a human would see:

• The full conversation history (for multi-turn tasks)

• Retrieved documents (for RAG)

• System prompts (for understanding constraints)

• Tool call results (for agentic workflows)

If a human needs it to judge quality, the evaluator needs it too.

Mistake 4: Not Providing Diverse Examples

If all your few-shot examples are “customer angry, agent apologizes,” the judge won’t know how to handle “customer confused, needs technical explanation.”

Cover the failure modes you actually see in production.

Mistake 5: Using Ready-Made Metrics Without Validation

ROUGE, BLEU, BERTScore, etc. sound professional, but they might not correlate with your actual goal.

Before using any metric, validate it against human judgment on your specific task. If high ROUGE doesn’t mean “users are happy,” don’t optimize for ROUGE.

Mistake 6: Using 1-5 Scales Instead of Binary Pass/Fail

Wrong:
Score: 3.2 out of 5

Right:
Score: 0 (Fail)
Critique: “Response didn’t provide a timeline or next steps.”

Why it matters: A score of 3.2 is ambiguous. Is that good enough to ship? Should you fix it? Binary forces clarity. Either it passes your quality bar or it doesn’t. Scoring on a float scale (0.0-1.0) has the same problem - it leaves room for interpretation instead of forcing a clear decision.

When Should I Use Similarity Metrics (BERTScore, ROUGE, etc.)?

Short answer: Only for specific, narrow tasks where semantic overlap actually matters.

When They Work

Summarization: ROUGE measures how much of the source content appears in the summary. If your task is “don’t miss key facts,” ROUGE helps.

Translation: BLEU checks n-gram overlap with reference translations. Works when there’s a narrow acceptable output space.

Retrieval accuracy: BERTScore compares semantic similarity between retrieved chunks and expected documents.

When They Fail

Open-ended generation: Your AI agent says “I’ve refunded order #12345. You’ll see the credit in 3-5 days.” Reference says “Refund processed for order #12345, expect 3-5 business days.” Different words, same meaning. ROUGE fails.

Tone and helpfulness: Similarity metrics don’t measure if the tone was appropriate or if it actually helped the user.

Business outcomes: High similarity doesn’t mean the customer is satisfied, the sale closed, or the task completed.

The Rule

If your success criterion is “output should be semantically similar to the reference answer,” use similarity metrics.

If your success criteria are “user achieved their goal,” use app-level evaluators grounded in outcomes.

Advanced Metric Designs

Now let’s handle the hard cases: multi-turn conversations, complex workflows, and agentic systems.

Evaluating Multi-Turn Conversation Traces

A single-turn eval checks one input and one output. Multi-turn evals check entire conversations.

Challenges:

• Context must carry across turns

• Errors compound (one bad response derails the rest)

• You need to catch the first upstream failure, not downstream symptoms

Strategy:

1. End-to-end task success: Did the agent accomplish the user’s goal by the end?

2. Turn-by-turn checks: Evaluate each exchange individually

   • Did turn 3 maintain context from turn 1?

   • Did turn 5 escalate when the user got frustrated?

3. Failure attribution: When something breaks, find the first turn where it went wrong

Example (customer support conversation):

Turn 1:

User: “I need to return order #12345”

Agent: “Sure, I can help with that. What’s the reason for the return?”