In most AI projects, such as the financial assistant I am working on, there is a constant battle between Retrieval-Augmented Generation (RAG) and Context-Augmented Generation (CAG). Should you implement a heavy RAG architecture up front that might not even work, or does CAG get the job done? For example, in our financial assistant system, we ultimately decided to use RAG only when we really HAVE to, because it introduces zigzag retrieval patterns that require dozens of queries per operation, increasing latency.

Also, while building Brown, my writing agent, I hit another wall. Brown needs to ingest massive amounts of research to anchor its writing process. At 180,000 input tokens, the Gemini API became entirely unreliable.

I faced constant timeouts, disconnections, and infrastructure breakdowns. Huge context windows suffer from API reliability and infrastructure stability issues, as well as performance degradation. But the thing is, I didn’t want to overcomplicate my solution with a RAG layer, so I started looking around for other solutions.

Most engineers face this painful tradeoff when working with large documents. You can stuff everything into the context window, but performance degrades quickly. This causes context rot, which happens when attention degrades over long contexts and earlier information loses its influence [1], [2].

Alternatively, you can build a RAG pipeline. But that requires maintaining vector databases, chunking strategies, and retrieval evaluation infrastructure.

Even the tools we use daily, like Claude Code or Cursor, rely on summarization-based context compression that loses critical information. I just wanted to dump my research into one file and get good answers without the infrastructure breaking. Recursive Language Models (RLMs) solve this exact problem [3].

RLMs use an inference-time pattern that treats your input as an external environment the model interacts with programmatically. You do not need chunking infrastructure or embedding pipelines. The model writes code to explore, filter, and recursively process your data on demand.

This approach scales the effective input and output lengths of LLMs. Researchers tested RLMs up to 10 million tokens across GPT-5 and Qwen3-Coder, showing they easily outperform base models [3]. Base model performance degrades as a function of input length and task complexity, while RLM performance scales with less degradation.

RLMs are also a model-agnostic inference strategy, meaning they work with any model you choose.

However, this architecture has honest downsides you must consider. The inference cost has high variance due to differences in trajectory lengths. The system suffers from code fragility, meaning that if the model writes buggy code, the entire reasoning chain fails.

Errors in sub-calls can compound through the recursive tree, propagating hallucinations. Sequential sub-calls also create latency bottlenecks. This makes RLMs best suited for deep thinking applications rather than real-time chat.

To understand how we bypass these infrastructure limits, we need to examine the specific programming trick that keeps the model’s memory clean.

Here is what you will learn about this pattern:

• The mechanism that keeps massive documents outside the context window.

• The orchestration loop that drives programmatic data exploration.

• The specific use cases where this pattern outperforms retrieval systems.

• A practical method to approximate this behavior using Claude Code.

If You Want To Go Deeper Into Production AI (Product)

Patterns like RLMs show that the real challenge isn’t the model, but the infrastructure and systems around it, called the harness. If you want to master that harness, check out my Agentic AI Engineering course, built with Towards AI.

34 lessons. Three end-to-end portfolio projects. A certificate. And a Discord community with direct access to industry experts and me.

Rated 5/5 by 300+ students. The first 6 lessons are free:

Start here

The REPL Trick That Keeps Your Context Window Clean

RLMs introduce a simple core idea. Do not feed the document into the model’s context window. Instead, load it as a variable in a persistent programming environment and let the model write code to interact with it [4].

The model never sees your 10-million-token document directly. In a traditional agent, the prompt goes into the model, completely blowing up your context window. In an RLM, the context stays outside as an external variable, and the model receives only a symbolic handle to it.

The system initializes a Read-Eval-Print Loop (REPL), which is a persistent interactive programming environment where variables and state persist across iterations [3].

The root model receives only metadata, such as the total character count and data structure. It also receives instructions on how to access the REPL. The model then writes code to peek into, filter with regex, chunk, or summarize the data.

When the model identifies a sub-task, it uses a specific primitive such as llm_query(prompt, chunk) to spawn a fresh, isolated worker sub-model [3]. The system pauses, executes this sub-call, and returns the result to the root model’s REPL.

Variables persist across these REPL turns. The model aggregates findings into a buffer, building the response progressively across iterations. Once confident, it calls FINAL(answer) to stop the recursive loop and return the response [5].

RLMs essentially perform context engineering on autopilot. Traditional context engineering requires you to carefully curate what goes into the context window through retrieval and compression [1]. RLMs automate this by letting the model itself decide what to extract, filter, and process.

Costs and performance stay intact because the model filters the input context without explicitly seeing it. By writing Python scripts, the model processes only the relevant portions through sub-calls. Only constant-size metadata about execution results is appended to the root model’s history, keeping its context window small and clean.

Understanding this mechanical loop allows us to map the pattern directly to production harness engineering.

Turn Any Agent Into a Plan-Execute-Validate Machine

RLMs are an inference-time orchestration pattern that maps directly to production harness engineering. If you have built agent systems, you already know the components: a planning loop, tool execution and validation [7]. RLMs formalize this into a programmable, recursive architecture.

A robust RLM harness uses a multi-tiered architecture. The root controller is a frontier model that acts as the project manager. It plans the reasoning process, writes code, and coordinates execution, but never directly interacts with tools or the full document [8].

Worker sub-models are cheaper, faster models spawned via an operation such as llm_query() to handle specific, localized sub-tasks. This reduces overall costs while maintaining high quality. The aggregation layer is the REPL environment that combines recursive step results into a final structured response via persistent variables.

This setup naturally follows the plan-execute-validate mapping. In the plan phase, the root controller reviews the query, creates a reasoning plan, and decides how to decompose the problem. It might plan to regex-filter a codebase, chunk a document, or batch sub-calls for parallel analysis.

In the execute phase, the model translates the plan into code. It writes Python scripts, issues llm_query() calls, and spawns worker sub-models for parallel execution in isolated REPL environments. External tools, like web search, are provided ONLY to worker sub-models, keeping the root model’s context perfectly clean.

After execution, the system enters the validation phase, where results feed back as observations. The root model assesses accuracy, launches verification sub-calls, and handles errors by dynamically adjusting its plan. If the Python code fails, the error traceback is yielded back to the model as an event.

This allows the model to adapt and fix its code on the next turn. The cycle repeats until the model calls FINAL(answer).

Deploying this in the real world requires strict production guardrails. You must configure maxIterations to cap the number of REPL turns, typically between 10 and 50. You need maxDepth to limit the recursive stack depth, where a depth of 1 is usually sufficient.

You also need maxStdoutLength to truncate REPL output returned to the model to prevent context overflow. Finally, permission gating is required to provide sandboxed execution with explicit approval for sensitive operations.

Neither Claude Code nor OpenAI Codex uses true RLM patterns. They rely on summarization-based context compression, file-system state tracking and progressive disclosure techniques [9]. This creates a succession of agents connected by prompts and file state, rather than maintaining a persistent REPL environment with programmatic sub-calls.

With this architecture in place, we can identify the specific real-world scenarios where this pattern outperforms traditional data processing.

Four Scenarios Where RLMs Beat Traditional Approaches

RLMs are best suited for deep thinking applications that require accuracy, multi-step reasoning, and reliability over massive contexts. They are not suited for real-time, low-latency chat applications.

The first scenario is parsing large files without building retrieval infrastructure. Instead of building a hybrid index with vector and graph search, you keep everything in one file or directory and use an RLM agent to extract information on demand.

We can view the relationship between RAG and RLMs as a spectrum. For simple cases, RLMs replace RAG entirely, removing the need for chunking and embeddings. For advanced scenarios, RLMs complement retrieval beautifully.

You use semantic search to find your first pool of candidates, write the results to disk as cached short-term memory, and use an RLM to query that refined dataset on demand.

The retrieval narrows the haystack, and the RLM reasons deeply over what is left. I use this exact workflow for my research, dumping everything into a massive text file and using an RLM to extract relevant information.

The second scenario is complex software engineering and codebase comprehension. RLMs ingest massive codebases containing millions of tokens to answer questions about architecture, map dependencies, and perform reviews.

The RLM paper tested this on LongBench-v2 CodeQA using Qwen3-Coder with a Python REPL. The model writes code to break down the codebase, launches sub-queries to smaller language models, and aggregates findings [3].

The third scenario is enterprise legal and financial analysis. RLMs provide consistent interpretation across thousands of contracts, case files, and policies that would overwhelm a standard context window. They also excel at financial audits and due diligence by tracing, validating, and reasoning through massive financial datasets.

The fourth scenario is deep research and information synthesis. RLMs synthesize research across thousands of files by programmatically filtering, chunking, and summarizing. They enable knowledge graph exploration and multi-hop reasoning over large document dumps.

At scale, RLMs become both more accurate and cheaper than standard long-context approaches. They avoid paying for n-squared attention over massive contexts by having the model process only relevant slices via sub-calls. In all these scenarios, the RLM pattern succeeds because it treats the LLM as a project manager that decides what to look at and delegates sub-tasks to workers.

Knowing these optimal use cases helps us approximate the pattern using tools you likely already have installed.

Build a Naive RLM SKILL in Claude Code

Claude Code does not natively use the RLM pattern. It relies on summarization-based context compression, file-system state tracking, and progressive disclosure. However, you can approximate RLM behavior using Claude Code’s existing harness features to build a naive RLM SKILL.

First, you set up the environment by having the SKILL load the target file or directory as a reference. Instead of feeding it into the context window, it writes the file path and metadata to a prompt for the root agent.

Second, the root Claude Code agent receives only this metadata and a set of instructions for how to interact with it. It uses its Explore subagent type
to examine the data structure, identify relevant sections, and plan its approach.

Third, the SKILL uses Claude Code’s Agent tool to spawn subagents. Each subagent receives a focused prompt to read specific lines and extract mentions, returning a condensed summary of a few thousand tokens. This mirrors the RLM pattern of spawning isolated sub-calls that process slices of the input.

Finally, the root agent collects these subagent results. It aggregates them into a coherent answer and decides whether more exploration is needed or whether to finalize the output.

Here is what this naive RLM SKILL looks like as a SKILL.md file:

---
name: rlm-research-analyzer
description: "Analyze large research files by treating
  them as an external environment. Instead of stuffing
  content into context, the model explores, decomposes,
  and recursively processes the data through subagents."
---

# Analyze Large Research Files Using the RLM Pattern

## Step 1 — Initialize the environment

Accept the target file path as an argument. Do NOT read
the file into context. Instead, run a Bash command to
collect metadata:

wc -l <file_path>   # total lines
wc -c <file_path>   # total bytes
head -5 <file_path>  # short prefix

Write the metadata and file path to a temporary prompt
file at <working_dir>/rlm_prompt.md. The root agent
receives ONLY this metadata, never the full content.

## Step 2 — Plan the exploration

Read rlm_prompt.md. Based on the metadata and prefix,
decide how to decompose the file. Use an Explore
subagent to scan the file structure:

- Identify section boundaries, headings, or delimiters
- Estimate which regions are relevant to the query
- Produce a ranked list of target ranges to process

## Step 3 — Delegate to worker subagents

For each target range, spawn an Agent subagent with a
focused prompt:

"Read lines {start}-{end} of {file_path}. Extract all
findings related to {query}. Return a summary under
2000 tokens."

Launch multiple subagents in parallel when ranges are
independent. Write each subagent's output to
<working_dir>/slice_{n}.md.

## Step 4 — Aggregate and finalize

Read all slice files. Synthesize the findings into a
single coherent answer. If gaps remain, return to
Step 3 with new target ranges. Otherwise, write the
final output to <working_dir>/answer.md and present
it to the user.

Notice how the four steps map directly to RLM primitives. Step 1 mirrors REPL initialization, where the data becomes an external variable rather than context input. Step 3 replaces the theoretical llm_query() operation with Claude Code’s Agent tool. Step 4 mirrors the FINAL() call that terminates the recursive loop.

This naive approximation lacks several critical features. It has no true REPL persistence, as Claude Code subagents do not share a persistent variable space. The filesystem serves as a proxy for REPL state, but it is slower and less elegant.

It also lacks sandboxing, as Claude Code runs directly in your environment. Then you miss out on configurable guardrails like max_iterations and max_output_chars, requiring manual limits instead. You get the idea.

Still, I’ve been using a similar technique in all my current projects: instead of stuffing the research into a file, I dump everything into a dir and link everything together in an index.yaml file that contains URIs to all the files, plus metadata such as the title and a 1-2 sentence summary of each source. Like this, through the index.yaml file, Claude Code can efficiently navigate the whole research dump token through progressive disclosure.

My structure looks something like this:

research/
├── index.yaml
├── file_1.md
├── file_2.md
├── ...
└── file_N.md

Also, the only out-of-the-box implementation I found is within the DSPy framework.

The naive SKILL is a useful thought exercise and a practical first step. For production use, you should reference the DSPy framework’s dspy.RLM module.

What’s Next

RLMs represent a fundamental shift in how we process large inputs. We are moving from asking how to fit data in the context window to asking how we let the model interact with it programmatically. This is a great thought exercise on integrating specialized inference-time functionality into your harness.

As models get better at writing code and REPL environments become more sophisticated, the boundary between the model and its infrastructure will blur. The model does not just use tools, it writes the tools on the fly to solve the specific problem in front of it.

Your next practical step is to experiment with our SKILL or with the DSPy framework’s dspy.RLM module on a real problem. Point it at a large codebase you need to understand or a research corpus you need to synthesize. Start with something you have been using RAG or context stuffing on, and see whether the RLM approach is more effective.

But here is what I’m wondering:

How have you been passing large files, such as deep research results or books, to your agents so far? RAG, CAG or other creative techniques?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

Whenever you’re ready, here is how I can help you

If you want to go from zero to shipping production-grade AI agents, check out my Agentic AI Engineering course, built with Towards AI.

34 lessons. Three end-to-end portfolio projects. A certificate. And a Discord community with direct access to industry experts and me.

Rated 5/5 by 300+ students. The first 6 lessons are free:

Start here

Not ready to commit? Start with our free Agentic AI Engineering Guide, a 6-day email course on the mistakes that silently break AI agents in production.

References

1. (n.d.). Effective Context Engineering for AI Agents. Anthropic. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

2. (n.d.). MIT’s new ‘recursive’ framework lets LLMs process 10 million tokens without context rot. VentureBeat. https://venturebeat.com/orchestration/mits-new-recursive-framework-lets-llms-process-10-million-tokens-without-context-rot/

3. Zhang, A. L., Kraska, T., & Khattab, O. (2025). Recursive Language Models. arXiv. https://arxiv.org/abs/2512.24601

4. (n.d.). Recursive Language Models: the paradigm of 2026. Prime Intellect. https://www.primeintellect.ai/blog/rlm

5. (n.d.). Why Recursive Language Models (RLMs) Beat Long-Context LLMs. Dextra Labs. https://dextralabs.com/blog/recursive-language-models-rlm/

6. Mansurova, M. (2026, March 30). Going Beyond the Context Window: Recursive Language Models in Action. Towards Data Science. https://towardsdatascience.com/going-beyond-the-context-window-recursive-language-models-in-action/

7. (2026, March 21). The Anatomy of an Agent Harness. LangChain Blog. https://blog.langchain.com/the-anatomy-of-an-agent-harness/

8. (2025, December 24). Building Effective AI Agents. Anthropic. https://www.anthropic.com/engineering/building-effective-agents

9. (2026, March 25). Effective Harnesses for Long-Running Agents. Anthropic. https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents

Images

If not otherwise stated, all images are created by the author.

When we stripped everything back to plain Python, simple API calls, and a custom ReAct engine, things finally worked. What we accidentally built was a harness featuring specialized financial tools, domain-specific guardrails, and purpose-built context engineering.

We did not know the term yet, but the lesson was clear. The model was never the problem. The system and infrastructure around it were.

Most engineering teams obsess over which model to use. They debate GPT-4o versus Claude Opus versus Gemini. They chase LLM benchmark scores and swap models, hoping for better results.

But the model is only half the equation. The system and infrastructure around it determine whether your agent actually works in production.

TerminalBench 2.0 proved this. Changing only the harness moved the DeepAgent from LangChain from outside the top 30 to the top 5 [1].

This is what usually happens. You have a powerful model. You gave it tools and a prompt. It works in demos.

But shipping it to production means solving problems the model cannot solve alone. You must bridge context windows, recover from failures, serve multiple interfaces, and manage state across sessions.

The solution is harness engineering. This is the discipline of building the infrastructure around the model so it can do useful work reliably. As Mitchell Hashimoto noted, harness engineering is the practice of engineering a solution every time an agent makes a mistake, ensuring it never makes that specific mistake again [2].

By the end of this article, you will learn:

• What an agent harness actually is.

• The core components powering production AI systems.

• How the planning loop dictates agent actions.

• The design principles behind an effective toolset.

• How to manage memory using the filesystem.

Before we look at all its components and how they fit together, we must first define what a harness actually is.

Your Path to Agentic AI Engineering for Production (Product)

Most engineers know the theory behind agents, context engineering, and RAG. What they lack is the confidence to architect, evaluate, and deploy these systems in production. The Agentic AI Engineering course, built in partnership with Towards AI, closes that gap across 34 lessons (articles, videos, and a lot of code).

By the end, you will have gone from “I built a demo” to “I shipped a production-grade multi-agent system with evals, observability, and CI/CD.” Three portfolio projects, a certificate to back them up in interviews, and a Discord community with direct access to industry experts.

Rated 5/5 ⭐️ by 300+ early students saying “Every AI Engineer needs a course like this” and that is “An excellent bridge from experimental LLM projects to real-world AI engineering.”

Start learning today. The first 6 lessons are free:

Enroll here

So... What the Heck Is a Harness?

While talking with Jonathan Gennick from Manning, he said that the first time he heard about the term “harness” was in the context of horses. Let me explain. A horse is powerful on its own, but useless for farming without a harness. The straps and reins let you direct its strength toward useful work. The same applies to LLMs.

The model has intelligence. But without tools, memory, state, guardrails, and orchestration, you cannot put it to work reliably.

LangChain offers the clearest definition. An agent equals a model plus a harness. The harness is every piece of code, configuration, and execution logic that is not the model itself [1].

A basic agent, as we know it so far, is just a model, a prompt, tools, and a planning loop. A harness extends this by adding memory systems, guardrails, advanced orchestration, context engineering, and multi-agent coordination.

Usually, it also includes a serving layer that connects the agent to various user interfaces, such as terminal apps, web dashboards, IDE plugins, and messaging apps like Telegram.

Ultimately, a harness is a term for building real software applications using LLMs or other models as the operating system. Applications like Claude Code, OpenCode, OpenClaw, and Codex are all harnesses. You could swap the model inside them, but the real engineering value lives in the harness itself.

This introduces three distinct levels of engineering. Prompt engineering crafts the instructions. Context engineering dictates what goes into the context window and when.

Harness engineering is the full application and infrastructure. It controls when context loads, which tools are available, which actions are allowed, and how failures are handled. Each level encompasses the previous one [3].

Now that you understand what a harness is, the next step is to explore the internal architecture and see how these pieces connect.

The Anatomy of a Harness

A complete harness consists of the LLM, tools, a planning loop, context engineering, a sandbox, memory, an orchestration layer, and a serving layer. In other words, everything that has been hovering within the AI space is finally falling into one beautiful system.

One of the most distinctive features of modern harnesses is the multi-surface architecture. OpenClaw serves the same agent across a command-line interface (known as TUI), a web UI, desktop apps, Slack and Telegram/WhatsApp through a centralized Gateway using a typed WebSocket protocol.

Codex evolved from a simple terminal tool to an App Server using JSON-RPC over standard input and output. OpenCode uses a Bun JS HTTP server where any client connects via HTTP, utilizing an Event Bus to broadcast results in real-time [4], [5], [6].

This architecture introduces challenges. Multiple messages arrive in parallel from different clients. Users ask questions while the model is still processing.

To solve this, systems use priority queues and message buses. OpenClaw uses a lane-aware FIFO queue to ensure only one active run per session while allowing parallelism across different sessions.

At the core of all this infrastructure, the filesystem is king. As the most foundational harness primitive, it enables durable storage, workspace management, multi-agent collaboration, and versioning.

You heard me right, there is no fancy vector database in place. With AI, we are going back to basics, and nothing is purer than the filesystem itself.

Every production harness uses the filesystem as its primary state mechanism [1].

You might wonder if this is just traditional orchestration like Airflow. It is different in three key ways. The agent loop is non-deterministic, context management is a first-class concern, and the programmer inside the loop is the LLM itself. It is common to add durability to the harness using tools such as Prefect, Temporal or DBOS that natively support dynamic pipelines rather than predefined, rigid DAGs.

Let us zoom in on the first and most fundamental component: the planning loop.

How the Agent Decides What to Do Next

The most common pattern for the planning loop is ReAct, which stands for Reasoning and Acting. The model receives the current state, reasons about what to do next, takes an action via a tool call, and observes the result. This cycle repeats continuously until a strict stopping condition is met [5].

Consider a concrete example. A user asks the agent to fix a failing test. First, the model reads the test output, reasons that the import path is wrong, and edits the file through a tool.

Second, it re-runs the tests, sees a new type mismatch error, and fixes it. Third, it runs the tests again.

They pass, the model reasons the job is done, and it stops. The harness orchestrates this loop, while the model reasons and picks actions.

When tasks are too complex for a single agent, harnesses use orchestrator-worker patterns. The orchestrator decomposes a task, delegates subtasks to specialized workers, and aggregates the results.

In OpenCode, a dedicated task tool spawns subagents. Each subagent gets its own session, context window, and restricted tool set [7].

For tasks that span multiple context windows, Claude Code implements Ralph Loops. This harness mechanism intercepts the model’s attempt to exit via a hook. It reinjects the original prompt in a clean context window, forcing the agent to continue against a completion goal using the state persisted on the filesystem [1].

While automating my business with agents, I learned a hard lesson about orchestration. I initially built five specialized agents, each handling one step.

I eventually found that a single agent with memory and smart context engineering outperformed the whole swarm. Always start with one well-harnessed agent before reaching for multi-agent complexity.

Here is a deep dive into planning:

While the planning loop decides the next step, the agent still needs a way to interact with its environment.

The Tools That Let Agents Act

This interaction happens through a specific toolkit designed for autonomous execution.

First, Bash is a general-purpose tool. The agent can run any shell command to execute tests, linters, or builds. This gives the model code execution capabilities so it can design its own tools on the fly rather than being constrained by fixed options.

For example, the agent runs Python code and executes it through python -c "...", generates a script and runs it through python main.py or runs your code as python -m my_module.main.

Second, specialized filesystem tools handle common operations like reading, writing, editing, and searching. Doing file operations via Bash is slow and error-prone.

Specialized tools include safety checks. For instance, a read tool enforces absolute paths and line limits, while an edit tool validates the uniqueness of replacement strings.

Third, state management tools track session-scoped tasks. These give the agent working memory within a single session. For example, OpenCode has ToDoAdd and ToDoRead tools that add/read tasks from a queue to keep track of the plan it has to execute.

Finally, orchestration tools launch subagents with their own isolated prompts and context windows, such as OpenCode’s task tool or Claude Code’s agent tool.

Feedback loops are the most important principle around tooling. Boris Cherny, the creator of Claude Code, noted that giving the model a way to verify its work improves quality by two to three times. For example, OpenCode integrates the Language Server Protocol (LSP) to get real-time code definitions and diagnostics.

Undefined variables and type errors are fed back to the LLM for immediate correction. These tools do not act on the world. They feed vital information back to the planning loop.

Harnesses also enforce tool access control. In OpenCode, the planning agent cannot call edit tools. This prevents exploratory agents from accidentally modifying your code [5].

Here is a deep dive into tool calling:

Once the agent has its tools, it needs a secure place to use them. In production, this requires strict isolation.

Where Agents Run

Agents execute code, and that code can fail, crash, or delete all your files. I know I want my precious notes protected. Sandboxes isolate agent execution so failures do not affect the host system or other agents. The cherry on top is that they also enable horizontal scaling across parallel environments.

There is a strict tradeoff between security and capability. Not every harness uses the same approach. Codex uses a hard sandbox.

Each task runs in an isolated cloud container preloaded with the repository. This provides maximum safety, but the agent cannot access the host filesystem [6].

Conversely, OpenClaw uses a soft sandbox. The workspace is the default working directory. This grants maximum capability but introduces more risk.

OpenClaw deliberately avoids hard sandboxing to preserve full filesystem access. Most production harnesses sit somewhere between these extremes, depending on the trust model.

When you submit a task to Codex, the harness spins up a fresh cloud container. The agent works inside this container to read files, run tests, and install packages.

It cannot touch your local machine. When the job finishes, the results are extracted, and the container is destroyed.

Along with security, a major benefit of cloud sandbox environments is that they give the agent access to powerful computing resources. For example, if you want to train a model using a GPU, you can ask the agent to implement and run a training pipeline hosted in a sandbox powered by a GPU.

This is similar to manually SSHing to different VMs and running the code manually there. Based on the same principles, you can easily spin up multiple cloud sandboxes and run your agents in parallel.

On the other side of the spectrum, you can also run sandbox environments locally through Docker containers or isolated processes, similar to what Cursor does. Super useful when you want to try something out and give the agent full permissions to avoid having to supervise it.

While sandboxes provide a safe space for execution, they are ephemeral by design.

Memory Is Just the Filesystem

To survive across sessions and context windows, every harness manages state across three distinct memory layers. The first layer is the filesystem. This is the long-term memory.

It is durable and persistent, surviving across sessions. This is where progress files, git history, and session transcripts live.

The second layer is the RAM. This is the short-term memory, also known as the working memory. It holds the conversation history and tool results during an active session. It is fast but volatile.

The third layer is the context window. This is what the model actually sees. It is the strictest constraint, as everything the model knows about the current task must fit here.

The harness orchestrates the dynamics between these layers. On the read path, the harness selectively loads relevant state from the disk into the RAM.

It then assembles the context window using context engineering techniques such as compaction, progressive disclosure, and just-in-time retrieval. On the write path, the harness persists important state back to the disk after processing.

OpenClaw enforces a strict invariant that memory is always flushed to disk before being discarded from context. Rehydration is treated as a tool-shaped action, where the agent searches and then retrieves specific data, rather than dumping everything into the context window [8].

Context engineering makes this possible. When token counts exceed ninety percent of the limit, OpenCode automatically summarizes the conversation. Codex assembles prompts from multiple sources and exploits prompt caching.

Anthropic recommends using structured note-taking files and sub-agent architectures to isolate context [5], [6], [9].

In Anthropic’s long-running agent pattern, an initializer agent creates a script, a progress file, and a feature list. The coding agent reads the git logs and progress files at the start of each session and updates the progress file as it progresses.

The beauty? There is no database or vector store. It is just the filesystem [10].

Here is a deep dive into memory:

Now that you have seen all the pieces, from planning and tools to sandboxes and memory, the question is what this means for how you build software.

What’s Next

We are witnessing a new way of building software. Instead of software engineers building traditional frontend and backend applications, the next generation of production software will be harnesses. Harness engineering is merging software engineering with AI, moving it one level up [3].

Popular tools like Claude Code are just the beginning. In the long run, no company will want to depend entirely on proprietary harnesses. Even open-source solutions like OpenCode will not cover every specific use case.

Companies will inevitably build their own. As we experienced at ZTRON, custom systems and infrastructure are what finally make an agent work in production.

However, we must be honest about current limitations. Memory still breaks across long sessions. Validation loops still miss edge cases. Furthermore, orchestrating hundreds of parallel agents on shared codebases remains an open research problem.

Harness engineering is real engineering. Your harness becomes its own product with its own bugs, its own drift, and its own maintenance burden.

What’s your opinion? Do you agree, disagree, or is there something I missed?

Leave a comment

Enjoyed the article? The most sincere compliment is to share our work.

Share

Go Deeper

Go from zero to production-grade AI agents with the Agentic AI Engineering self-paced course. Built in partnership with Towards AI.

Across 34 lessons (articles, videos, and a lot of code), you’ll design, build, evaluate, and deploy production-grade AI agents end to end. By the final lesson, you’ll have built a multi-agent system and a capstone project where you apply everything you’ve learned on your own.

Three portfolio projects and a certificate to showcase in interviews. Plus a Discord community where you have direct access to other industry experts and me.

Rated 4.9/5 ⭐️ by 300+ students saying “Every AI Engineer needs a course like this.”

Learn more

Not ready to commit? We also prepared a free 6-day email course to reveal the 6 critical mistakes that silently destroy agentic systems. Get the free email course.

References

1. LangChain. (2026, March 21). The Anatomy of an Agent Harness. LangChain Blog. https://blog.langchain.com/the-anatomy-of-an-agent-harness/

2. Hashimoto, M. (2026, March 25). My AI Adoption Journey. Mitchell Hashimoto. https://mitchellh.com/writing/my-ai-adoption-journey

3. Bouchard, L. (2026, March 25). What Harness Engineering Actually Means. What’s AI by Louis-François Bouchard. https://youtube.com/watch?v=zYerCzIexCg

4. Govindarajan, V. (2026, March 21). OpenClaw Architecture Part 1 - The Agent Stack. The Agent Stack. https://theagentstack.substack.com/p/openclaw-architecture-part-1-control

5. Abboud, M. (2026, March 17). How Coding Agents Actually Work: Inside OpenCode. Moncef Abboud. https://cefboud.com/posts/coding-agents-internals-opencode-deepdive/

6. ByteByteGo. (2026, March 26). How OpenAI Codex Works. ByteByteGo. https://blog.bytebytego.com/p/how-openai-codex-works

7. Anthropic. (2025, December 24). Building Effective AI Agents. Anthropic. https://www.anthropic.com/research/building-effective-agents

8. Govindarajan, V. (2026, March 24). OpenClaw Architecture Part 3: Memory and State Ownership. The Agent Stack. https://theagentstack.substack.com/p/openclaw-architecture-part-3-memory

9. Anthropic. (2025, October 22). Effective Context Engineering for AI Agents. Anthropic. https://www.anthropic.com/engineering/effective-context-engineering

10. Anthropic. (2026, March 25). Effective Harnesses for Long-Running Agents. Anthropic. https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents

Images

If not otherwise stated, all images are created by the author.

It is 2026. People across the industry still mix up words like workflows, agents, tools, and multi-agent systems. Beyond terminology, this confusion has led to massively overengineered solutions.

Teams jump to multi-agent architectures because it sounds impressive and helps raise money. In reality, a simple workflow would have been faster to build, cheaper to run, and easier to debug. The result is bloated systems, wasted tokens, and debugging nightmares.

Our goal is to provide a clear mental model of what architecture to choose for your AI project: workflows vs. single agents vs. multi-agent systems.

from Towards AI has been working on this exact problem with his clients and distilled his decision framework into two YouTube videos: Stop Overengineering: Workflows vs AI Agents Explained and From Workflows to Multi-Agent Systems: How to Choose. He allowed me to take that framework and turn it into this article. Kudos to Louis-François!

This decision framework is a spectrum from simple to complex that tells you exactly what to build based on your actual constraints. The goal is to stay as far left on the complexity spectrum as possible while still solving your problem.

Here is what you will learn:

• The fundamental difference between an agent and a workflow.

• How to use the complexity spectrum to make architecture decisions.

• When to rely on simple workflows for predictable tasks.

• Why a single agent with tools is often enough for dynamic problems.

• The exact breaking points that justify moving to a multi-agent system.

To apply this spectrum effectively, you must first define the terms. Here are the core misconceptions that lead to bad architecture decisions.

Before we continue, a quick word from the Decoding AI team. ↓

Go Deeper: Your Path to Agentic AI for Production

Most engineers know the theory behind agents, context engineering, and RAG. What they lack is the confidence to architect, evaluate, and deploy these systems in production. The Agentic AI Engineering course, built in partnership with Towards AI, closes that gap across 34 lessons (articles, videos, and a lot of code).

By the end, you will have gone from “I built a demo” to “I shipped a production-grade multi-agent system with evals, observability, and CI/CD.” Three portfolio projects, a certificate to back them up in interviews, and a Discord community with direct access to industry experts like and me.

Rated 4.9/5 ⭐️ by 300+ early students saying ”Every AI Engineer needs a course like this” and that is ”an excellent bridge from experimental LLM projects to real-world AI engineering.”

Start learning today

↓ Now, back to the article.

Clarifying the Confusion: Not Everything Is an Agent

The first major misconception is that every LLM application is an agent. The key difference is autonomy. In a workflow, you control the flow.

You decide the steps and their order. In an agent, the model controls the flow. It decides what to do next based on the goal you give it.

If you can write down the exact sequence of steps in advance, you are building a workflow. You are not building an agent.

The second misconception is that tools are agents. A tool is a capability. It can be a calculator, a database query, a web browser, a validator, or an API call.

It can even be another LLM. An agent is the decision maker who chooses which tools to use and when.

If someone tells you they built a multi-agent system, but it is actually one model calling ten different APIs, that is not multi-agent. That is a single agent with ten tools.

This distinction matters. It defines how you architect, debug, and scale your system. It drives your core architecture choice between a workflow, a single agent with tools, or multiple agents.

The Complexity Spectrum: A Mental Model for Architecture Decisions

To make this architecture choice easier, we use a complexity spectrum. It is a slider going from the most control to the most autonomy. Your goal is to stay as far left as possible while still solving the problem.

Level 1 represents workflows. Here, you chain multiple LLM calls together in a predefined sequence. You control every step.

Level 2 represents a single agent with tools. The model makes decisions about what to do next. You have one decision maker and multiple capabilities.

Level 3 represents multi-agent systems. Here, you have multiple decision makers who need to coordinate with each other.

The core principle is straightforward. Move right on this spectrum only when you absolutely have to. Each step to the right increases costs, latency, and debugging complexity.

More LLM calls mean more tokens, more traces to follow, and more places where things can go wrong.

In practice, start simple and escalate only where things break. Write a prompt first. Test it.

Implement it with minimal complexity. Measure the results. Add what is missing.

If the model lacks information, add retrieval. If it needs calculations, add a tool. Only when you genuinely need autonomous decision-making should you reach for an agent.

Even then, start with one. The best AI systems are the simplest ones that reliably solve the problem. That usually means starting with workflows.

When a Workflow Is the Right Answer

Workflows are the right answer when your steps are known and stable. If the process is largely the same each time, regardless of input, a workflow is almost always the best choice.

Workflows win because they are predictable. They are easy to test because you can write unit tests for each step. They are easy to debug because you can trace exactly what happened when something goes wrong.

They are also cheap because you are not burning tokens on the model, figuring out what to do next.

Consider a support ticket system. A ticket comes in. You classify it.

You route it to the right team. You draft a response from templates and context. You validate it against the policy.

Finally, you send it. Each step might involve an LLM call, but the model does not need to decide whether to classify before routing. That is always the order.

Building this as an agent adds overhead without adding capability.

Do not underestimate workflows. They are not limited to simple sequential chains. They can include routing to pick different models based on input.

They can use parallel execution with majority voting to aggregate answers. They can also use generator-evaluator loops where one LLM generates and another validates until quality criteria are met. They can even leverage tools in designs like the orchestrator-worker. These patterns handle complex tasks without any agent overhead.

If you can write down the exact sequence of steps in advance, like a recipe, it is a workflow.

When a Single Agent with Tools Wins

Sometimes the order of work is not fixed. You genuinely cannot write down the steps in advance. This happens when the path changes depending on what you discover along the way.

Maybe the first API call fails, and you need to try an alternative. Maybe the retrieved data is incomplete, and you need clarification. This is what agents handle well.

When is an agent worth the risk? Anthropic offers a useful framework. Agents make sense when the task is complex enough to need autonomous decisions and delivers real value.

Critically, the cost of errors and the cost of discovering those errors must be low. This is why AI coding agents are great. A human reviews the code before production, so mistakes are cheap to fix.

A purchasing agent who accidentally buys the wrong hardware makes an expensive error. You must match your architecture to your error tolerance [3].

The rule is to always start with one agent. A single agent with tools works best when tasks are tightly coupled and mostly sequential. It works well when global context matters, meaning step one affects step five.

It is also ideal when you need fewer than twenty tools and face strict budget or latency constraints.

Take a marketing content platform from Louis-François’s client work at Towards AI. The client wanted AI-assisted content generation for emails, text messages, and promotional materials. Their initial specification called for a multi-agent setup with over a dozen specialized agents.

They wanted an orchestrator, a request analyzer, a content generator, and many others. On paper, it looked clean with specialists doing specialist work.

A single agent was the right call. The tasks were tightly coupled and sequential. The template choice affects the content.

Personalization depends on both content and contact data. Splitting this across multiple decision makers creates information silos and handoff errors. They did not need parallelism.

The flow was to plan, generate, validate, and fix if needed.

The key insight is that tools can be smart. A tool can have its own system prompt and use a different model. The validation tool can use its own LLM with instructions to catch errors.

The text message tool can treat character limits as deterministic engineering constraints instead of prompting problems. You get specialists, but you keep one brain to maintain context and make final decisions.

This results in a system that is faster to build, cheaper to run, and easier to debug. You get the same capabilities without the coordination overhead.

The Tool Count Problem: When One Agent Isn’t Enough

As your tool list grows, tool selection gets harder. This is one of the main ways agent systems quietly break down. It is also one of the clearest signals that splitting into multiple agents might be worth it.

Every tool has a name, description, and schema that the model needs in context to use correctly. The more tools you add, the more of your context budget you burn before the agent even starts thinking about the actual task. You also have to add system instructions, a few-shot examples, retrieved documents, and conversation history on top of that.

A single agent tends to work best with roughly 10 to 20 tools. Past that threshold, tool selection degrades. The agent has to choose among too many options in an already packed context.

This mechanism is known as context rot. LLM performance measurably degrades as context grows, well before hitting the advertised limit. Two forces drive this issue.

First, more context means more noise competing for the model’s attention. Second, models suffer from loss in the middle. They tend to attend more to the beginning and end of their context, underweighting information in the middle.

As your tool schemas and instructions pile up, the model gets worse at picking the right tool.

Managing context can reduce history and retrieved content, but not the tool schema load. Those definitions must always be there. The only approach that actually reduces how many tool definitions the model sees per call is splitting across agents.

If one agent sees only email tools and another only sees validation tools, each call stays smaller. Tool selection gets easier. Once you split tools across agents to keep calls small, you enter multi-agent territory.

When Multi-Agent Is Actually the Right Call

Specific reasons justify multiple agents, not because the architecture sounds impressive. There are four legitimate reasons to go multi-agent. First, you need true parallelism where tasks are genuinely independent and run simultaneously.

Second, you face context overload where instructions and tools degrade performance. Third, you need modularity to connect with third-party agent systems you do not control. Fourth, you have hard separation requirements like security boundaries or sensitive data handling.

Consider the professional article generation system that Louis-François and I built as one of the projects for our Agentic AI Engineering course. We started with a single agent for research and writing but had to pivot because the two phases have fundamentally different needs.

The research phase is exploratory and dynamic. It needs flexibility and broad tool access across web search, video transcription, and document processing. The agent searches, reads, pivots based on what it finds, and iterates based on human feedback.

The writing phase is constrained and deterministic. It needs focused constraints, consistent style enforcement, and iterative refinement against fixed rubrics.

These agents communicate through explicit artifacts. The research agent produces a structured markdown file that the writer agent consumes as context. There is no complex runtime orchestration.

It is just a sequential handoff with a clear contract between them. Each agent has its own optimized context without the bloat of carrying the other’s tools.

If you do go multi-agent, we recommend the plan-and-execute combined with the orchestrator-worker pattern. You do not want everyone talking to everyone. One orchestrator maintains the main context and delegates specific tasks to worker agents.

Then, it synthesizes the results. This prevents the information silos that kill multi-agent systems.

Multi-agent systems can simplify individual contexts and enable specialization. However, they increase coordination costs. You will face more token usage, added latency, more failure points, and handoff complexity.

Only accept those costs when you hit a real constraint that simpler architectures cannot solve.

To Wrap Up

To build reliable AI applications, you must stay as far left on the complexity spectrum as possible while still solving your problem.

Keep these key takeaways in mind:

• Not every LLM application is an agent, and not every tool is an agent.

• Always start with workflows because they are predictable, cheap, and testable.

• Use one agent when the path cannot be predetermined, but keep the tool count manageable.

• Move to multi-agent architectures only when you hit a real constraint like true parallelism or context overload.

Each step right on the spectrum increases cost, latency, and debugging complexity. The simplest system that reliably solves the problem is always the best system.

💡 If you want a step-by-step framework to help you decide what architecture to pick for your next project, Louis-François and the Towards AI team put together a free cheatsheet that walks you through the decision process from workflows to multi-agent systems.

What’s your opinion? Do you agree, disagree, or is there something I missed?

Leave a comment

Enjoyed the article? The most sincere compliment is to share our work.

Share

Go Deeper

Go from zero to production-grade AI agents with the Agentic AI Engineering self-paced course. Built in partnership with Towards AI.

Across 34 lessons (articles, videos, and a lot of code), you’ll design, build, evaluate, and deploy production-grade AI agents end to end. By the final lesson, you’ll have built a multi-agent system and a capstone project where you apply everything you’ve learned on your own.

Three portfolio projects and a certificate to showcase in interviews. Plus a Discord community where you have direct access to other industry experts and me.

Rated 4.9/5 ⭐️ by 300+ students saying “Every AI Engineer needs a course like this.”

Learn more

Not ready to commit? We also prepared a free 6-day email course to reveal the 6 critical mistakes that silently destroy agentic systems. Get the free email course.

Images

If not otherwise stated, all images are created by the author.

Welcome to the AI Evals & Observability series: A 7-part journey from shipping AI apps to systematically improving them. Made by busy people. For busy people.

AI Evals is the topic most AI engineers know they should invest in, but do not know where to start. I remember struggling with this myself.

I did not know how to properly integrate evals into my app until I understood there are three core layers: optimization during development, regression testing before merging, and production monitoring on live traffic. Once that clicked, everything else fell into place.

I did not know how to build LLM judges and evaluators that I could actually trust and use. Every guide I found either hand-waved the details or dumped a generic “helpfulness” metric and moved on. Instead, I needed evaluators grounded in my actual business requirements.

I did not know how to gather custom datasets without wasting too much time. I tried generating hundreds of synthetic test cases up front, but the real unlock came from learning how to organically grow a high-quality dataset from production data, starting small and letting the error-analysis flywheel do the heavy lifting.

The information was scattered across blog posts, talks, and vendor docs. Most of it focused on isolated techniques without showing how everything connects. I built this series as the structured, end-to-end guide I wish I had.

This 7-lesson series breaks it all down from first principles. By the end, you will know how to integrate AI evaluations that actually track and improve your product's performance. No vibe checking required.

The series follows a natural progression. You start by understanding where evals fit. Then, you build the dataset.

Next, you design and validate the evaluators. Finally, you handle specialized domains like RAG and see how it all works in production.

You can read front-to-back for the full journey. Alternatively, jump to the lesson that matches your current pain point. Each lesson stands on its own but references the others.

Without more yada, yada, here are the 7 lessons of the series:
(Scroll down to find more about each lesson individually.)

1. Integrating AI Evals Into Your AI App

2. Build an AI Evals Dataset from Scratch

3. Generate Synthetic Datasets for AI Evals

4. How to Design Evaluators

5. How to Evaluate the Evaluator

6. RAG Evaluation: The Only 6 Metrics You Need

7. Lessons from 6 Months of Evals on a Production AI Companion

Everything is completely free, without any hidden costs, thanks to our sponsor, Opik ↓

Opik: Open-Source LLMOps Platform (Sponsored)

This AI Evals & Observability series is brought to you by Opik, the LLMOps open-source platform used by Uber, Etsy, Netflix, and more.

We use Opik daily across our courses and AI products. Not just for observability, but as our end-to-end evaluation harness, all from the same platform.

This series teaches you how to build evals from scratch (custom datasets, LLM judges, optimization loops, and production monitoring), while Opik gives you the platform to run everything at scale.

Here is how we use it:

• Custom LLM judges: Build evaluators by defining your criteria, adding a few-shot examples, and running them across hundreds of traces automatically.

• Run experiments, compare results: Test different prompts, models, or parameters from your AI app side by side. Opik scores each variant with your evaluators and shows you which one wins.

• Plug evaluators into production: The same LLM judges you design for offline testing run on live traces too. Set up alarms when scores drop below your threshold so you catch regressions before users do.

Opik is fully open-source and works with custom code and with every popular AI framework or tool (including OpenClaw). You can also use the managed version for free (with 25K spans/month on their generous free tier):

Try Opik for free

↓ Now, let’s move back to the article.

Lesson 1: Integrating AI Evals Into Your AI App

To build a reliable system, you first need to know where evaluation fits into the development lifecycle.

Most teams start by “vibe checking” their AI app. They manually test a few inputs and eyeball whether the outputs look right. That works for the first version.

But the moment you start adding features, onboarding real users, or trying to improve existing capabilities, vibe checking collapses. This first article gives you the holistic map of where AI Evals fit, so you never feel lost again.

Here is what you will learn:

• The three core scenarios where evals matter: optimization during development, regression testing before merging, and production monitoring on live traffic.

• The difference between guardrails and evaluators. Confusing them leads to gaps in your system.

• The minimum viable tech stack required to start: a custom annotation tool and an LLMOps platform.

Go to Lesson 1

Lesson 2: Build an AI Evals Dataset from Scratch

Once you understand where evals fit, the next step is gathering the data required to measure performance.

You cannot evaluate what you cannot measure. You cannot measure without data. Most teams either skip this step entirely or fire off a generic prompt to create 100 test cases and call it done.

This article teaches the error analysis framework. It is a practical flywheel that turns 20-50 real production traces into a growing, high-quality evals dataset.

Here is what you will learn:

• The error analysis flywheel: sample traces, label manually, build evaluators iteratively, perform error analysis, and create specialized evaluators.

• Why one “benevolent dictator” should own labeling consistency across your team.

• How to graduate from generic to specialized evaluators as your understanding deepens.

Go to Lesson 2

Lesson 3: Generate Synthetic Datasets for AI Evals

Production traces alone have limits. You need traffic to get data, and that traffic rarely covers every scenario. What about before you have users?

What about rare failure modes you have never seen in production? Yet! Synthetic data solves the cold start problem and fills coverage gaps.

Here is what you will learn:

• Why you should generate only inputs, not outputs, and let your real app produce the outputs.

• How to think in dimensions like persona, feature, scenario, and input modality to avoid mode collapse.

• Tester agents for simulating multi-turn conversations.

• The reverse workflow for RAG: generate questions from your knowledge base, not the other way around.

Go to Lesson 3

Lesson 4: How to Design Evaluators

You have the dataset. Now you need evaluators who can actually tell you whether your app is working. This is where most teams make their biggest mistake.

They grab a generic helpfulness metric off the shelf and call it done. This article teaches you how to design evaluators grounded in your actual business requirements.

Here is what you will learn:

• The evaluation harness: the infrastructure that automates running evaluators across your dataset.

• When to use fast, deterministic code-based evaluators versus flexible, nuanced LLM judges.

• Common design mistakes

• Advanced designs for multi-turn conversations and agentic workflows.

Go to Lesson 4

Lesson 5: How to Evaluate the Evaluator

You built an evaluator. It says everything is great. But is it?

An evaluator that validates every output is worse than no evaluator at all. It gives you false confidence. This article teaches you how to validate your evaluator against human judgment and close the gap when they disagree.

Here is what you will learn:

• The iterative refinement loop: measure alignment, diagnose disagreements, adjust few-shot examples, and re-measure.

• Dealing with non-determinism: why LLM judges give different answers on the same input, and how to stabilize them.

Go to Lesson 5

Lesson 6: RAG Evaluation: The Only 6 Metrics You Need

After mastering general evaluators, you can apply these principles to specific architectures like RAG.

RAG evaluation feels overwhelming because everyone proposes different metrics. But it does not have to be complicated. This article proves that there are exactly three variables in any RAG system: Question, Context, and Answer.

There are exactly six possible relationships between them. That is it. Every RAG metric maps to one of these six relationships.

Here is what you will learn:

• The three RAG variables and six exhaustive relationships.

• Tier 1: Retrieval metrics. If retrieval is broken, nothing else matters.

• Tier 2: The three core RAG metrics you always need.

• Tier 3: When core metrics cannot explain the failure.

Go to Lesson 6

Lesson 7: Lessons from 6 Months of Evals on a Production AI Companion

Theory and isolated metrics are useful. But the ultimate test is running this entire system on live user traffic.

The first six articles teach you how to build the system. This final article shows you what it looks like after six months of running it in production.

Written as a guest post by , Senior Data Engineer at Workpath, it shares the real lessons. We cover what worked, what failed, and what they wish they had known from the start.

Here is what you will learn:

• The three observability problems most teams hit: falling for generic metrics, skipping manual annotation, and not treating AI agents as data products.

• How to use Opik’s architecture, including traces, spans, threads, and prompt versioning, for production monitoring and evals.

• How to reverse-engineer evaluation criteria from real traces instead of guessing upfront.

Go to Lesson 7

How to Take the Course?

After completing these seven articles, you will have the complete mental model for AI Evals. You will understand everything from strategy to production.

As the course is 100% free, with no hidden costs or registration required, taking it is a no-brainer.

Each lesson is a free article hosted on the Decoding AI Magazine.

Just open each lesson in the order provided by us, and you are good to go:

1. Integrating AI Evals Into Your AI App

2. Build an AI Evals Dataset from Scratch

3. Generate Synthetic Datasets for AI Evals

4. How to Design Evaluators

5. How to Evaluate the Evaluator

6. RAG Evaluation: The Only 6 Metrics You Need

7. Lessons from 6 Months of Evals on a Production AI Companion

Each lesson will guide you through the required steps.

Enjoy!

Now What?

After completing these lessons, if you want the information to stick, you have to put everything into practice by building a cool project!

I am sorry to say there is no other way to make learning worthwhile. Pick one problem and get your hands dirty with a project.

💡 Want to share your work on my socials with my 140k+ audience? If you build a project you are excited about, I will be too. Trust me! I love seeing people build cool stuff. To share it, you can contact me here.

See you next Tuesday.

Paul Iusztin

What’s your opinion? Do you agree, disagree, or is there something I missed?

Leave a comment

Enjoyed the article? The most sincere compliment is to share our work.

Share

Go Deeper

Go from zero to production-grade AI agents with the Agentic AI Engineering self-paced course. Built in partnership with Towards AI.

Across 34 lessons (articles, videos, and a lot of code), you’ll design, build, evaluate, and deploy production-grade AI agents end to end. By the final lesson, you’ll have built a multi-agent system and a capstone project where you apply everything you've learned on your own.

Three portfolio projects and a certificate to showcase in interviews. Plus a Discord community where you have direct access to other industry experts and me.

Rated 4.9/5 ⭐️ by 300+ students — “Every AI Engineer needs a course like this.”

Learn more

Not ready to commit? We also prepared a free 6-day email course to reveal the 6 critical mistakes that silently destroy agentic systems. Get the free email course.

Thanks again to Opik for sponsoring the series and keeping it free!

If you want to monitor, evaluate and optimize your AI workflows and agents:

Try Opik for free

Images

If not otherwise stated, all images are created by the author.

Most agents fail silently. They work well in demos but drift unpredictably in production. Costs spike with no clear explanation.

Behavior becomes erratic, and every release feels risky. Ultimately, teams end up stuck in PoC purgatory, unable to ship, debug, or trust their own system.

The root cause is almost never the model. It is subtle system design mistakes that individually look small but compound into production disasters.

To fix this, together with , we created a diagnostic framework for six specific mistakes that cause agentic systems to break in production. Each has a clear problem, a reason why it happens, and a proven fix. Once you know what to look for, you can trace most production failures back to one of these patterns.

The first and most common failure starts right at the input level, where engineers mishandle the context window.

Mistake #1: Treating the Context Window as an Afterthought

When something breaks, the instinct is to add more context. Engineers add more rules, more history, more tools, and more examples. The assumption is that if the model sees everything, it will behave better.

But this turns the context window into a dumping ground instead of a carefully scoped working memory. As the context grows, the model starts to ignore instructions and apply constraints inconsistently. It hallucinates more and drifts across runs.

Latency spikes and costs compound. This is the lost in the middle problem. Many teams respond by splitting one giant prompt into dozens of smaller ones.

But that introduces its own problems, such as more LLM calls, higher latency, and harder debugging.

💡 Treat the context window as a scarce resource.

Every LLM call should have one clearly scoped job. You must curate context aggressively by selecting, compressing, and pruning before every call. Move persistence into a memory layer.

The context window holds only what matters for the next decision, and everything else lives in memory, which you write to and read from continuously.

As a rule of thumb, start with a single prompt. If it works, stop. If it fails, do not jump to agents.

Introduce a small number of specialized steps and tune until you hit the balance. Context engineering is about deliberate selection.

Once the context window is secure, the next trap is overengineering the architecture before the problem demands it.

Mistake #2: Starting with Complicated Solutions

You have a clear problem, so you immediately reach for multi-agent architectures or heavy frameworks. You build RAG pipelines, hybrid retrieval, multiple databases, or adopt new protocols like MCP. You do this not because the problem demands it, but because it feels like the right way to build serious AI.

Every layer adds a hidden tax. You get more dependencies, higher latency, higher costs, and harder debugging. Complexity compounds operational pain.

Teams end up spending months building infrastructure and shipping nothing.

At our startup, ZTRON, we built a multi-index RAG system. We had OCR pipelines, separate embedding pipelines, hybrid retrieval, and agentic RAG loops.

It worked, but simple queries took 10 to 15 seconds. Costs climbed, and debugging was a nightmare.

When we finally asked if we actually needed all this, the answer was no. Our data fit within modern context windows. We replaced agentic RAG with cache-augmented generation (CAG) for most workflows.

This gave us fewer LLM calls, lower latency, fewer errors, and an easier system to debug.

Start with the simplest solution that could work. Prove the core task works first. Only add memory, tools, retrieval, or multiple agents when the problem demands it.

Production-grade AI is built by engineers who ship simple systems first and scale complexity intentionally.

Earning complexity often means realizing that you do not need an agent at all, which brings us to the third mistake.

Mistake #3: Building Agents When a Workflow Will Do

Predictable tasks like data ingestion, summarization, or report generation need predictable execution. That is a workflow. Open-ended tasks like deep research or dynamic decision-making under uncertainty may need autonomy.

Agents handle these open-ended scenarios. Most teams treat predictable problems as if they need agents. When you use an agent for a structured task, you pay for autonomy you do not need.

You get unpredictable behavior, variable latency, higher token usage, and inconsistent outputs. The system works 80% of the time and fails when it matters most.

Workflows and agents are not binary choices. They sit on a spectrum known as the autonomy slider. More autonomy buys flexibility but costs predictability, cost control, and debuggability.

You must set the slider intentionally.

Adopt a workflow-first approach. Start with prompt chaining, routing, parallelization, or an orchestrator-worker pattern. Introduce agents only when the system must autonomously plan, explore unknown paths, or recover from failures dynamically.

For vertical AI agents, use a hybrid approach. Route known patterns to workflows and open-ended requests to agents.

Whether you use a workflow or an agent, you must handle the data they produce, which exposes a flaw in how engineers process outputs.

Mistake #4: Fragile Parsing of LLM Outputs

You ask the model for something structured, and it responds with something that looks structured. You parse it with regex, string splitting, or custom logic. It works in staging.

Then one day, a missing comma or different bullet style crashes production. LLMs are non-deterministic. Even with identical prompts, output can drift due to context changes, model updates, or variations in tool outputs.

Fragile parsing is a time bomb. Many teams respond by prompting the model to output JSON. That is better than free-form text, but it still is not a contract.

You still get missing keys, wrong types, and drifting nested fields.

Stop treating LLM outputs like text and treating them like data. Define a schema, enforce it at generation time, validate at runtime, and fail fast when wrong. Use Pydantic as the bridge between probabilistic generation and deterministic code.

But only use structured outputs when structure is required. If you only need a plain string, accept a string and keep schemas shallow and minimal.

If you have secured your context, simplified your architecture, chosen the right autonomy, and enforced output schemas, you are ready to build an agent. However, many teams still fail by omitting actual planning from their loops.

Mistake #5: Forgetting Agents Need Planning

You give a model tools, let it pick one, feed the tool output back, and repeat. At a glance, it looks agentic, but it is just a workflow with randomness. The system is reacting to the last tool output, not driving toward a goal.

Without embedded planning, the loop cannot decompose tasks into meaningful steps. It cannot evaluate progress or choose next actions intentionally. The result is random behavior, unnecessary tool calls, infinite loops, and shallow reasoning.

Copying ReAct or Plan-and-Execute from blog posts without adapting them to your domain makes it worse.

You must embed planning into the loop. Before calling a tool, require a reasoning step. Ask what the goal is, what the next best action is, and what evidence you need.

Add progress checks and stop conditions like max steps, token budgets, and escalation when stuck. Make planning use-case specific, because generic ReAct is not a product. Tailor planning to your tools, data, constraints, and failure modes.

Even a well-planned agent will degrade over time if you do not measure its performance continuously.

Mistake #6: Not Starting with AI Evals from Day Zero

You build features without tracking how well your AI behaves. You have no tests, no evaluation metrics, and no defined success criteria. Every new feature is a gamble, and teams silently ship regressions.

AI systems do not fail all at once. They decay. A prompt change, a new tool, or a model upgrade causes subtle behavior shifts.

Without evals, nobody can answer whether a change made the system better or worse. Teams get stuck relying on vibe evals, which are manual, gut-feel testing that does not scale. Many teams think they are doing evals, but rely on generic scores like helpfulness or 1-5 star scales.

A score of 3.7 helpfulness tells you nothing about what to fix.

Use evals as your north star. Define task-specific, binary metrics tied to real system behavior and business requirements from day one. Use evals to drive the optimization flywheel.

Integrate evals into your development workflow to catch regressions before users do.

Recognizing these six mistakes is the first step to escaping PoC purgatory.

Conclusion

These six mistakes are not exotic edge cases. They are the exact patterns that repeatedly break real agentic systems. Individually, they look small, but in production, they compound into disasters.

Each of these mistakes deserves a deeper breakdown with real examples and production-tested fixes. That is why we turned them into a free 6-day email course. We cover one mistake per day, with the exact patterns and solutions we use in production.

💡 If you want the complete breakdown, sign up here.

Otherwise, see you next Tuesday.

Paul Iusztin

What’s your opinion? Do you agree, disagree, or is there something I missed?

Leave a comment

Enjoyed the article? The most sincere compliment is to share our work.

Share

Go Deeper

Go from zero to production-grade AI agents with the Agentic AI Engineering self-paced course. Built in partnership with Towards AI.

Across 34 lessons (articles, videos, and a lot of code), you’ll design, build, evaluate, and deploy production-grade AI agents end to end. By the final lesson, you’ll have built a multi-agent system and a capstone project where you apply everything you’ve learned on your own.

Three portfolio projects and a certificate to showcase in interviews. Plus a Discord community where you have direct access to other industry experts and me.

Rated 4.9/5 ⭐️ by 300+ students saying “Every AI Engineer needs a course like this.”

Learn more

Not ready to commit? We also prepared a free 6-day email course to reveal the 6 critical mistakes that silently destroy agentic systems. Get the free email course.

Images

If not otherwise stated, all images are created by the author.

🧐 Everyone says you need AI evals. Few explain how to actually build them and answer questions such as…

How do we avoid creating evals that waste our time and resources? How do we build datasets and design evaluators that matter? How do we adapt them for RAG? ...and most importantly, how do we stop “vibe checking” and leverage evals to actually track and optimize our app?

This 7-article series breaks it all down from first principles:

1. Integrating AI Evals Into Your AI App

2. Build an AI Evals Dataset from Scratch

3. Generate Synthetic Datasets for AI Evals

4. How to Design Evaluators

5. How to Evaluate the Evaluator

6. RAG Evaluation: The Only 6 Metrics You Need ← You are here

7. Lessons from 6 Months of Evals on a Production AI Companion

By the end, you’ll know how to integrate AI evals that actually track and improve the performance of your AI product. No vibe checking required!

Let’s get started.

RAG Evaluation: The Only 6 Metrics You Need

In our previous article, we covered how to validate your AI judges. We measured agreement with human judgment and iterated until alignment was high. Thus, you can now deploy with confidence.

However, a specialized challenge exists that general-purpose grading tools do not fully address. Evaluating RAG systems introduces a third variable, specifically the retrieved context. With this new element comes a distinct set of failure modes requiring their own metrics.

I am currently building a financial personal assistant at the stealth AI startup I work for. The application runs heavily on RAG. It pulls financial data from Postgres and integrates with external services such as email, Customer Relationship Management (CRM) tools, and cloud drives.

When it came time to evaluate the system, building the dataset proved harder than choosing metrics. Fortunately, we had a domain expert on the team who manually tested the application from the start. Therefore, we translated all of that Quality Assurance (QA) work into our AI evals collection using the error analysis workflow from Article 2.

Evaluating RAG systems introduces a unique difficulty. Each data sample required the correct context to be loaded into the database. We solved this by coupling every test case with a Postgres SQL export.

This file contained documents, chunks, embeddings, and metadata. We injected it directly into the storage system. This effectively created a cache that bypassed the ingestion pipeline during evals.

Once the data was in place, implementing the core RAG metrics became straightforward. We used tools like Opik and foundational models like Gemini Pro as the LLM judge. We had the context, the query, and the answer, which is everything you need.

What surprised me was that not every capability needed this level of dissection. For our report generation feature, we expect an exact format with specific values pulled from the storage. Checking the final document against a ground truth served as a better proxy than tracing every retrieval step.

Sometimes assessing the destination matters more than checking the route.

RAG evaluation feels needlessly complex. Vendors have an incentive to make it difficult. Every framework ships with many metrics and a dashboard, making you feel like you need a PhD to know if your system works.

Underneath all the complexity, RAG systems possess exactly three core components. These are the Question (Q), the retrieved Context (C), and the generated Answer (A). Furthermore, with these elements, there are exactly six possible relationships between them. When your RAG system fails, it breaks along one of these six relationships every single time.

The beauty of this framework is its exhaustive nature. There are no hidden variables.

You do not always need to evaluate all six individually. For core conversational features, you need the primary metrics because there are many silent failure modes. However, for structured output tasks, an end-to-end check against expected results can be sufficient.

Here is what you will learn in this article:

• The only six relationships that exist in a RAG system.

• How to evaluate your retrieval step before looking at generation.

• The three core metrics every RAG application needs.

• Advanced metrics for diagnosing subtle hallucinations.

• How to match evaluation frequency and strictness to your domain.

• How to collect and prepare the data your evaluators need.

Before digging into the article, a quick word from our sponsor, Opik. ↓

Opik: Open-Source LLMOps Platform (Sponsored)

This AI Evals & Observability series is brought to you by Opik, the LLMOps open-source platform used by Uber, Etsy, Netflix, and more.

We use Opik daily across our courses and AI products. Not just for observability, but to design and run the exact RAG evaluators this article teaches. All from the same platform.

This article shows you how to evaluate RAG systems. Opik gives you the harness to run those evaluations at scale. Here is how we use it:

• Custom LLM judges with rubrics — Build the evaluators this article describes: define your criteria, add few-shot examples, and run them across hundreds of traces automatically.

• Run experiments, compare results — Test different prompts, models, or configurations side by side. Opik scores each variant with your evaluators and shows you which one wins.

• Plug evaluators into production — The same LLM judges you design for testing run on live traces too. Set up alarms when scores drop below your threshold so you catch regressions before users do.

Opik is fully open-source and works with custom code or most AI frameworks. You can also use the managed version for free (with 25K spans/month on their generous free tier):

Try Opik for free

↓ Now, let’s move back to the article.

The Only 6 RAG Evaluation Metrics That Can Exist

Jason Liu properly articulated the framework I am about to walk you through [1]. Since I wrote the LLM Engineer’s Handbook two years ago, I have watched many RAG evaluation tools emerge. They overcomplicate everything with proprietary metric suites.

Through all of that, I already internalized that only three variables matter in any RAG system. Testing the combinations between them is the only thing you should actually do. Jason Liu gave a clean, formal articulation to what I had in mind.

He nailed the structure and deserves the recognition for that.

Every RAG system has three variables. We define Q as the user’s question, C as the retrieved context, and A representing the generated answer. Thus, we use the notation X|Y to mean the quality of X given Y.

There are exactly six relationships between these variables:

1. C|Q (Context Relevance) asks if the retrieved context addresses the question. This measures your retriever, because if it pulls irrelevant passages, the generator cannot fix the issue.

2. A|C (Faithfulness) checks if the answer sticks to what is in the context. This measures your generator to see if the model hallucinated or stayed grounded in the documents.

3. A|Q (Answer Relevance) verifies if the response actually addresses the prompt. This is the end-to-end user experience metric. Even if the context is good and the reply is faithful, it must help the person asking.

4. C|A (Context Support) ensures the retrieved text contains everything needed to support every claim in the answer. This checks if the provided information was sufficient.

5. Q|C (Question Answerability) evaluates if the prompt can even be resolved with this context. This determines whether the system should attempt to reply at all.

6. Q|A (Self-Containment) asks if someone can infer the original question from reading the answer alone. This measures whether the output provides enough background to stand on its own.

This framework is exhaustive. Three components produce exactly six conditional relationships. There are no hidden factors.

Therefore, when your RAG system fails, one of these six metrics is broken.

Not all six relationships matter equally in every context. We organize them into three tiers. Let us start with retrieval metrics as the prerequisite foundation.

Tier 1: If Retrieval Is Broken, Nothing Else Matters

RAG is first and foremost a retrieval problem. If the search mechanism does not retrieve the right documents, nothing downstream can save you. The generator will either hallucinate or produce irrelevant answers based on whatever junk it received.

Before evaluating any of the six RAG relationships, you need to know if your retriever even works. You can use classic information retrieval metrics that measure how well you find relevant documents before generation starts. They are fast to compute and do not require LLM judges.

Hence, these measurements give quick feedback for tuning your retriever.

You must establish ground-truth labels to compute these metrics. For each query, you must know which text blocks are actually relevant. You can build this dataset using the reverse workflow presented in depth in Article 3.

As a quick recap, you start from your knowledge base of document chunks. Then, based on a set of closely related chunks, you generate realistic questions that can only be answered using that unique set of chunks.

Because the prompt derives from the source material, you know exactly which segment should be retrieved. This gives you a perfectly aligned ground-truth triplet: (question, answer, context). Thus, it becomes straightforward to check whether your search tool actually surfaces the right information.

There are four main metrics. Precision@K measures the fraction of the top K retrieved chunks that are actually relevant. If your retriever returns 5 chunks but only 2 are useful, your precision is 40%. Recall@K asks: of all the relevant chunks that exist in your entire corpus, how many did your retriever actually find in the top K? If the database has 4 chunks that could answer the question but you only retrieved 2 of them, your recall is 50%.

In addition, Mean Average Precision (MAP@K) averages precision across multiple queries, rewarding retrievers that consistently rank relevant chunks early. It works by computing precision at every position where a relevant item appears, then averaging those values. Here is a step-by-step example where the truly relevant items are A and C:

Average Precision for this query = (1.0 + 0.66) / 2 = 0.83. We only average the precision values at positions where a relevant item appeared (ranks 1 and 3). MAP@K then takes this score and averages it across all your queries.

Finally, Mean Reciprocal Rank (MRR@K) focuses on the position of the first relevant match. If the first relevant chunk appears at position 3, the reciprocal rank is 1/3; if it appears at position 1, it is 1/1. Higher is better.

Use these for daily development. These indicators are great for tuning embeddings and chunk sizes, while also being perfect for A/B testing retrieval strategies. No LLM is needed, making the process cheap and fast.

These numbers tell you if the search phase works, as illustrated in Image 3. The six RAG relationships tell you if the whole system functions properly, meaning you need both.

With the retrieval confirmed working, you can evaluate the generation step. Let us look at the three core RAG relationships that every system needs.

Tier 2: The Three RAG Metrics You Always Need

These three metrics directly assess how well your RAG system functions. Most evaluation frameworks prioritize these specific measurements. They map to the three most critical of the six relationships.

First, we have Context Relevance (C|Q). This checks if the retrieved text actually addresses the prompt’s information needs. Therefore, it measures your search component similar to the metrics from Tier 1, but only looking at the dynamics between the context and question, without any ground truth.

Suppose we have a query about recent payouts from Q4. A good example is when the retrieved data contains the user’s dividend payment records from Q4, which passes. On the other side, a bad scenario occurs when the system returns general information about how these distributions work and their tax implications.

This represents the most common RAG failure mode. In our financial assistant, this often happened when the search tool pulled educational content instead of actual account data.

Second, we have Faithfulness (A|C). This asks if the reply restricts itself to claims that can be verified from the provided text. Hence, it measures whether your generator hallucinates or not.

In our use case, a good example is when the source contains a CRM record showing a client meeting scheduled for portfolio rebalancing. If the response states exactly that, it passes. A bad example happens when the model adds hallucinated agenda items like tax-loss harvesting strategies, resulting in a failure.

Third, we have Answer Relevance (A|Q). This checks if the output directly addresses the specific query from the prompt. This serves as the end-to-end user experience metric.

A good example is when a person asks how much their investments grew last month. The reply provides the specific percentage change and absolute dollar amount. A bad scenario is when the text discusses general market performance without mentioning the actual account.

We measure all three metrics using LLM judges as designed in Article 4 and validated in Article 5.

These three metrics cover the most common failure modes. For specific domains and failure cases, we have to dig deeper into the next 3 metrics.

Tier 3: When the Core Metrics Can’t Explain the Failure

The last three metrics provide deeper diagnostic insights usually required in sensitive domains or use cases.

First, we have Context Support (C|A). This checks if the retrieved context contains all the information needed to fully back every claim in the response. While this sounds similar to Faithfulness (A|C), the direction is different. Faithfulness asks: “did the answer deviate from the context?” , where you look at the answer and check if it introduced claims that aren’t there. Context Support asks: “was the context sufficient to support the answer?”, where you look at the context and check if it actually contains everything the answer needs.

Here is a concrete example. Suppose the answer says your total Q4 dividend income was 2,340 across 5 holdings, with the largest payout from MSFT at 890. Now look at the context: it only contains the total dividend amount of $2,340. The per-holding breakdown is nowhere in the retrieved documents. The context was insufficient. It had the total but not the details. The LLM produced a plausible breakdown, but the context could not support it. This is low-context support.

Second, we have Question Answerability (Q|C). This asks if the user's question can even be resolved with the given information.

Suppose the user asks about crypto portfolio performance, but the retrieved documents only contain equity data. This makes the request unanswerable. The system should refuse rather than guess. This metric is important when you want to validate that your agent answers with “I don’t know” instead of confidently hallucinating an answer due to insufficient context.

In our financial assistant, this was important because some queries can only be resolved if the agent has permissions to access the right external tool first.

Third, we have Self-Containment (Q|A). This checks if someone can infer the original prompt from the reply alone.

A response stating your portfolio’s return is 12.4% stands alone. A reply stating just 12.4% does not. Prioritize this metric when outputs are forwarded via email, logged in CRM notes, or read without the original conversation.

You now know what to measure at each tier. Two questions remain. How often should you run each one? Which metrics deserve the most attention for your specific domain?

Matching Frequency and Strictness to Your Domain

Each tier maps to a different running frequency depending on how fast and cheap you can run the evaluations. It also depends on their overall impact on the system.

Start with Tier 1 on a daily basis. Implement fast retrieval metrics for everyday development and to tune your retrieval component. These are the cheapest to execute as they do not require LLM judges.

Furthermore, they provide quick feedback cycles. Use them for the improvement flywheel with synthetic data from day zero, focusing on these basic indicators before moving to more complex approaches.

Move to Tier 2 on a weekly basis. Implement the three primary RAG connections. These core metrics directly assess how well your system functions.

Use LLM-based grading for a more nuanced assessment of these interactions.

Incorporate Tier 3 on a monthly basis. Introduce advanced metrics when you need deeper insights. Run a full evaluation to identify prompts that the application should not be answering.

Here, we focused only on RAG-related measurements. However, this actually applies to any type of AI evals layer. You could implement Tier 1 checks in your CI/CD pipeline to execute on each commit.

You can trigger Tier 2 evaluations manually before merging your code from your feature branch. Finally, manually run Tier 3 metrics before major releases and strategic decisions.

There is another dimension to consider when choosing metrics for your use case, which is the good old domain.

Different domains require emphasis on distinct indicators. What matters most depends on the severity of the use case.

High-severity domains include finance, medical, and legal applications. In these fields, Faithfulness (A|C) and Context Support (C|A) are non-negotiable because every claim must be traceable. Answerability (Q|C) is also critical, meaning the application must refuse rather than guess.

Thus, you want precision over recall, which is the exact profile we use for our financial assistant.

Medium severity domains include customer support and technical documents. Answer Relevance (A|Q) leads here, as the output must be helpful and correct. Answerability (Q|C) helps you know when to hand off to a human, and you generally want recall over precision in retrieval.

Low-severity domains include research, writing, and content generation, where synthesis and creative reframing are expected. Context Relevance (C|Q) and Answer Relevance (A|Q) is primary, while Faithfulness (A|C) thresholds remain lower. The generator is supposed to add value beyond the raw text.

Therefore, you want high recall in the search phase to cast a wide net across sources.

You know what to measure, when, and what to prioritize. None of this works without the right data and infrastructure. Let us explore how to build the evaluation harness.

Building the RAG Evaluation Harness

RAG evaluation requires inputs, outputs, and the retrieved context. You need the full triplet.

The most common blind spot involves treating RAG testing like any other LLM assessment. Teams measure the final reply’s quality, but never capture what background data the generator actually worked with. Without that information, half the metrics in this article are impossible to compute.

Next, you should ground your RAG dataset in real human judgment. In our financial assistant, we had a domain specialist on the team who manually QA’d the application from the start. They ran queries, checked whether the right data was retrieved, and verified that the replies made sense.

We translated all of that manual work into our AI evals collection using the error analysis workflow from Article 2.

Also, building RAG datasets introduces a unique difficulty. Each test case needs the right documents, chunks, and embeddings available in the database. Otherwise, the search tool has nothing to work with.

Running the full ingestion pipeline for every evaluation run is slow and introduces variability.

We solved this by coupling each data point with a Postgres SQL export containing the relevant documents, chunks, embeddings, and metadata. We loaded this file directly into the storage system for each test, effectively creating a context cache. This made the process fast and reproducible.

We inject the records, run the query, evaluate the trace, reset the environment, and move to the next item. Image 7 illustrates these data preparation paths.

If you do not have enough production data or expert QA samples, you can create synthetic RAG evaluation sets. Use the reverse workflow from Article 3 by starting from your knowledge base. Use an LLM to extract key facts from specific passages.

Then, formulate realistic user questions that can only be answered using that exact text block.

Because the prompt derives directly from the source material, the input, expected retrieval context, and expected reply are perfectly aligned by construction. This gives you a complete ground-truth triplet. Furthermore, this technique is especially powerful for bootstrapping coverage across your entire document corpus.

Include unanswerable queries in your collection. Do not only formulate prompts that the application should resolve correctly. Instead, create scenarios where the context deliberately lacks the information needed, forcing the agent to refuse or say it does not know.

Without these negative examples, your testing suite is one-sided. Your evals will optimize for always attempting a reply, whereas adding them directly exercises the Answerability metric from Tier 3.

Next, if your RAG architecture integrates with external services, the retrieval path is not just a vector database search. Your agent needs to decide which tool to call first. Should it query Postgres, search the CRM, or check the user’s email?

The best retrieval metrics will not help if your model invoked the wrong data source entirely.

In our financial assistant, this was critical. A query about a client meeting should hit the CRM, not the transaction database. Therefore, we added code-based checks for tool selection alongside our RAG metrics.

Another important trick is to run separate graders per RAG dimension. Do not ask one LLM to evaluate context relevance, faithfulness, and answer relevance in a single prompt. Isolated checks with dimension-specific rubrics produce more consistent results than a unified evaluation.

Ultimately, you need to log specific data for every trace using tools such as Opik. Record retrieved chunks to see what the generator had access to. If faithfulness fails, check whether the reply used information that was not provided. Track metadata such as document IDs and scores, because when context relevance fails, you need to know which items ranked highest. This represents the same observability infrastructure from Article 1.

Next Steps

RAG evaluation is not complex. It is just three variables and six relationships. When your RAG system fails, one of these specific links is broken.

Fix that exact issue and ignore the complexity theater.

Start with Tier 1 retrieval checks as daily prerequisites. Add Tier 2 primary indicators weekly. Extend to Tier 3 when specific failure modes demand it.

Ultimately, match your evaluation priorities to your domain’s risk profile.

Next time you see a vendor dashboard with dozens of RAG metrics, map each one back to the six relationships. If an indicator does not clearly measure one of the core links, it is noise. Drop it and focus on what actually diagnoses failures.

Next up is the final piece in the series. We will explore real-world lessons from months of running evals on a production AI companion. We will discuss what worked, what failed, and what the team would do differently.

Also, remember that this article is part of a 7-piece series on AI Evals & Observability. Here’s what’s ahead:

1. Integrating AI Evals Into Your AI App

2. Build an AI Evals Dataset from Scratch

3. Generate Synthetic Datasets for AI Evals

4. How to Design Evaluators

5. How to Evaluate the Evaluator

6. RAG Evaluation: The Only 6 Metrics You Need ← You just finished this one

7. Lessons from 6 Months of Evals on a Production AI Companion

See you next Tuesday.

Paul Iusztin

What’s your opinion? Do you agree, disagree, or is there something I missed?

Leave a comment

Enjoyed the article? The most sincere compliment is to share our work.

Share

Go Deeper

Go from zero to production-grade AI agents with the Agentic AI Engineering self-paced course. Built in partnership with Towards AI.

Across 34 lessons (articles, videos, and a lot of code), you’ll design, build, evaluate, and deploy production-grade AI agents end to end. By the final lesson, you’ll have built a multi-agent system and a capstone project where you apply everything you've learned on your own.

Three portfolio projects and a certificate to showcase in interviews. Plus a Discord community where you have direct access to other industry experts and me.

Rated 4.9/5 ⭐️ by 300+ students — “Every AI Engineer needs a course like this.”

Learn more

Not ready to commit? We also prepared a free 6-day email course to reveal the 6 critical mistakes that silently destroy agentic systems. Get the free email course.

Thanks again to Opik for sponsoring the series and keeping it free!

If you want to monitor, evaluate and optimize your AI workflows and agents:

Try Opik for free

References

1. Liu, J. (2025, May 19). There Are Only 6 RAG Evals. jxnl.co. https://jxnl.co/writing/2025/05/19/there-are-only-6-rag-evals/

2. Grace, M., Hadfield, J., Olivares, R., & De Jonghe, J. (2026, January 09). Demystifying Evals for AI Agents. Anthropic Engineering Blog. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

Images

If not otherwise stated, all images are created by the author.

She’s deconstructing RAG with a production-first mindset, skipping the theoretical demos to focus on building for architectural reliability.

This one is packed. Let’s get into it 👀 ↓

The “Deer in the Headlights” Moment

I’ve navigated many shifts since the early days of the web, from monoliths to cloud-native microservices and SOAP to REST. But the AI wave felt different. I found myself in a “deer in the headlights” moment, completely unsure of what to learn or even where to start. Should I dive into neural network math, focus on model training, or master context engineering (AI moves quickly)?

Eventually, the path became clear when I realized my real value lay in applying AI to complex business problems. In an enterprise context, that led me straight to RAG. It isn’t just about the model, it’s also about the robust system you build around it. It felt like a return to architecture, a concrete problem to solve where using AI could make a profound difference. However, as I started building, I hit a second roadblock...

Why Most RAG Tutorials Didn’t Help Me Learn RAG

Most RAG tutorials optimize for one outcome: getting an answer out of a model as quickly as possible. That’s fine for demos. It’s a poor way to learn how RAG systems behave in production.

I’m not new to building production software. I’ve spent decades shipping and maintaining systems where debuggability, operability, and failure modes matter. What’s new to me here is RAG, not the discipline of building systems that survive contact with reality. While learning RAG, I wanted to internalize the constraints I’d eventually face anyway: inspectability, idempotent ingestion, debuggable retrieval, and controllable generation. That meant resisting framework-managed chains and owning the control flow early, even if it slowed me down.

This post documents how I’m teaching myself RAG by building a production-grade system in deliberate phases, using frameworks as utilities rather than architecture.

That approach was heavily influenced by, and indeed, inspired by Paul Iusztin’s From 100+ AI Tools to 4: My Production Stack, especially this idea:

AI frameworks are good utilities. They should not dictate the architecture or control flow of your system.

That became my guiding principle.

Before we continue, a quick word from the Decoding AI team. ↓

Go Deeper: Your Path to Agentic AI for Production

Most engineers know the theory behind agents, context engineering, and RAG. What they lack is the confidence to architect, evaluate, and deploy these systems in production. The Agentic AI Engineering course, built in partnership with Towards AI, closes that gap across 34 lessons (articles, videos, and a lot of code).

By the end, you will have gone from “I built a demo” to “I shipped a production-grade multi-agent system with evals, observability, and CI/CD.” Three portfolio projects, a certificate to back it up in interviews, and a Discord community with direct access to industry experts and Paul Iusztin.

Rated 4.9/5 ⭐️ by 290+ early students — ”Every AI Engineer needs a course like this” and ”an excellent bridge from experimental LLM projects to real-world AI engineering.”

Start learning today

↓ Now, back to the article.

The Architecture

Before diving into the details, here is the end-to-end architecture of the RAG system. This diagram serves as a reference model, and we’ll walk through each layer and the production considerations that shaped these choices.

Phase 1. Ingestion: Own the Data

What I built: a pipeline that discovers files → loads documents → normalizes text → chunks → embeds → stores everything in Postgres.

From experience building production systems, ingestion pipelines are where complexity quietly accumulates if they lack idempotence, i.e., the ability to safely re-run without ending up in an inconsistent state, such as duplicate data, partial updates, or stale artifacts. The same applies to traceability, i.e., the ability to trace exactly what happened, to which data, and when. I assumed the same risks would apply here.

What I didn’t account for was how the nature of debugging would differ so vastly from what I was used to in the past. It wasn’t just about emitting log and error information at the right places anymore. A bad chunk doesn’t throw an exception, it just hallucinates an answer three steps later.

Single database, many uses

Instead of introducing a separate vector database, I used Postgres + pgvector. Chunks, embeddings, and metadata live together. That decision buys me a lot:

• I can inspect ingestion results with plain SQL

• I can join vectors with relational metadata

• I can reproduce retrieval behavior outside the application

That inspectability matters when you’re still learning, and having less infrastructure to maintain pays off long after.

Frameworks as utilities, not architecture

I use LangChain’s document loaders (TextLoader, PyMuPDFLoader) for format handling. But the control flow is explicit and mine:

for file_info in discover_files(folder_path):

    raw_docs = load_document(file_info.file_path)

    clean_text = normalize_text(raw_docs)

    chunks = chunk_text(clean_text, chunk_size=512)

    embeddings = await embed_chunks(chunks)

    await save_to_postgres(file_info, chunks, embeddings)

Each step is isolated. Each step can be logged, rerun, or replaced independently. When something breaks, I debug my code, not a framework-managed chain. For instance, during my initial tests, I used PyPDFLoader for the document loading phase. When I inspected the chunking, I realised the chunks had incorrect spaces due to kerning (e.g., ”P r e - C h u n k”). This was easy to address just by swapping PyPDFLoader with PyMuPDFLoader, which handled the complex layouts better.

Idempotence and safe re-runs

I mentioned earlier that pipelines break down when they lack idempotence. Here’s how I addressed it.

Every file’s contents are hashed. If the content hash matches what’s already stored, the file is skipped, no wasted compute, no risk. If the content has changed, its old chunks and embeddings are completely removed before the new ones are written. The database never ends up with a mix of old and new states for the same source.

During development, it makes experimentation safe. For instance, I can tweak chunk sizes, swap embedding models, or change preprocessing logic, then re-run the entire pipeline and trust the result. Without this, every experiment would mean manually cleaning up the database first, or worse, not realizing stale data was still there, silently affecting retrieval quality. More importantly, though, in production, it makes the pipeline resilient to failure. If ingestion crashes halfway through, I can simply restart it. Files already processed are skipped, and the rest pick up where they left off. No manual cleanup, no risk of corruption.

Phase 2. Retrieval: Make Failure Visible

Retrieval is where the quality of your results is determined, which makes debugging discipline more important than clever algorithms.

What I built: query preprocessing → embedding → similarity search → optional reranking.

Most LangChain tutorials show you how to build a RAG pipeline as a “chain,” i.e., a single call where the framework retrieves context, sends it to the LLM, and returns the answer. I chose not to do that. Consistent with the architecture philosophy above, retrieval is an explicit phase, and every step in the retrieval pipeline is an explicit function call I control and invoke directly:

async def retrieve(query: str, top_k: int = 5, rerank: bool = False):

    processed_query = preprocess_query(query)

    query_embedding = embed_query(processed_query)

    results = await search_similar_chunks(query_embedding, top_k)

    if rerank:

        results = rerank_results(query, results, top_k)

    return RetrievalResponse(query=query, results=results)

Keeping retrieval explicit makes failures legible. When an answer is wrong, I can tell whether the issue came from:

• query preprocessing

• embedding quality

• recall

• ranking

Because vectors live in Postgres, I can reproduce retrieval behavior with SQL alone.

That inspectability is invaluable when learning.

Retrieval → Generation boundary

This is the boundary where many RAG systems start to blur failure modes. But they are fundamentally different problems.

Retrieval, including reranking, decides what context is allowed to reach the model. It is a search problem. It fails by missing relevant information (poor recall) or burying it in noise (poor precision).

Generation decides what the model does with the provided context. It is a reasoning problem. It fails by misinterpreting the context, hallucinating facts, or ignoring instructions.

Keeping this boundary explicit helps you immediately diagnose which problem you effectively have. If the answer is wrong but the context contains the truth, you fix the prompt. If the context is missing the truth, you fix the search.

Phase 3. Generation: Treat the LLM as an Unreliable Dependency

What I built: context formatting → LLM invocation with retries → response assembly.

LLMs fail in ways traditional dependencies don’t. They are non-deterministic, occasionally unavailable, and can return plausible but wrong outputs. I treated the model as an unreliable dependency from day one, something to isolate, observe, and swap, not something to trust implicitly.

Swappable LLMs via a factory

A simple factory pattern makes experimentation cheap:

def get_llm():

    if provider == “openai”:

        return OpenAIChat(...)

    if provider == “gemini”:

        return GeminiChat(...)

Switching providers requires only configuration changes. Call sites don’t care. This is exactly where frameworks like LangChain shine: as an abstraction layer. They handle the messy API differences between providers so that OpenAIChat and GeminiChat can expose the same interface to your application. Using them here makes swapping models trivial, without letting them dictate your control flow.

Explicit orchestration over chains

Generation is intentionally step-by-step:

async def generate_answer(request):

    retrieval_response = await retrieve(query=request.query, ...)

    context_text = format_docs(retrieval_response)

    messages = get_rag_prompt().format_messages(

        context=context_text,

        question=request.query,

    )

    llm = get_llm()

    ai_message = await _invoke_llm_with_retry(llm, messages)

    return GenerateResponse(answer=ai_message.content, ...)

I avoided using LangChain’s expression language (LCEL) or runnable abstractions to build this flow. While powerful, they can hide what’s happening. Explicit orchestration is easier to debug, instrument, and reason about, especially while learning. This resonated with me even more since I’m used to a hands-on approach where I can write code and truly understand how the logic flows.

Retries are operational, not semantic

LLM calls fail for mundane reasons: transient network issues, provider-side throttling, or brief outages. I treat those as operational failures, not model behavior, and handle them explicitly.

from tenacity import retry, stop_after_attempt, wait_exponential

@retry(

    stop=stop_after_attempt(3),

    wait=wait_exponential(multiplier=1, min=1, max=10),

)

async def _invoke_llm_with_retry(llm, messages):

    return await llm.ainvoke(messages)

Retries don’t make the model correct, they make the system resilient.

Phase 4. Serving: Thin Adapters, Shared Core

What I built: two interfaces over the same RAG core: a REST API and an MCP server.

In many RAG implementations, the retrieval logic is tightly coupled to the web framework (e.g., defined inside a FastAPI route). This makes it hard to test the logic in isolation or reuse it in different contexts (like a CLI or an evaluation script).

Instead, I treated my RAG system as a standalone library. The core function ‘generate_answer’ takes a pure Pydantic object and returns one. It knows nothing about HTTP, headers, or JSON.

This architecture allowed me to treat serving as a thin adapter pattern.

Adapter 1: REST API (FastAPI)

The REST adapter serves traditional software systems that need deterministic access to the retrieval layer. This includes web applications, backend services, internal tooling, evaluation pipelines, and automation jobs. These are environments where the caller decides exactly when and how the capability should be invoked.

The web layer itself does no extra work. It merely deserializes JSON, calls the core, and serializes the result.

@router.post(”“, response_model=GenerateResponse)

async def query(request: GenerateRequest) -> GenerateResponse:

    return await generate_answer(request)

Adapter 2: MCP Server (Capability Interface for Tool-Using LLMs)

Exposing the same core through the Model Context Protocol (MCP) transforms my RAG pipeline from an application-bound feature into a standardized capability.

MCP standardizes how capabilities are exposed to tool-using LLMs, regardless of whether the caller is a chat assistant, a coding copilot, or an autonomous agent.

I’m used to abstraction via careful refactoring, and it didn’t take long to understand that MCP was just another way of achieving this in the context of AI.

MCP-compatible clients such as Claude Desktop, Cowork, or Cursor can connect to the server and invoke the query_rag tool directly. This allows the underlying LLM to ground its responses in private data without requiring custom integrations, plugins, or connector logic.

Direct tool access is useful, but the MCP interface becomes far more valuable as agents adopt skills to carry out knowledge work and other multi-step tasks. For example, a “Market Research Skill” might combine web search, financial data lookup, and document retrieval. By exposing my RAG system as an MCP Tool, it becomes a standardized block that these skills can easily include in their workflows, without needing custom code.

@mcp.tool()

async def query_rag(query: str, top_k: int = 5, rerank: bool = True) -> dict:

    request = GenerateRequest(query=query, top_k=top_k, rerank=rerank)

    response = await generate_answer(request)

    return response.model_dump()

Both interfaces share the same core logic, thus avoiding duplication. Serving is an adapter problem, not a RAG problem.

Data lineage & traceability

Traceability isn’t new. Long before LLMs, production systems relied on lineage and identifiers to make failures debuggable. LLM non-determinism makes that discipline more important, not less.

Debugging RAG systems almost always means reasoning backward, from an answer, to retrieved chunks, to embeddings, and finally to source files.

In practice, this meant persisting identifiers at every step. Retrieved results carry chunk IDs forward. Generation logs include the IDs of the chunks used as context. When an answer looks wrong, I can trace it deterministically back to its source.

Without lineage, every bad answer looks like a model problem. With it, failures become diagnosable and fixable.

Vendor-neutral observability

This isn’t RAG specific. It’s the same observability discipline I’ve applied in other production systems. I deliberately kept it vendor-neutral, following a pattern I’ve used before to keep core logic decoupled from tooling.

Beyond tracing execution paths, tools like Opik let me reason about operational realities: latency per phase, token usage, and cost per request. Being able to see which model was invoked, how many tokens were consumed, and where time was spent turns performance and cost from assumptions into measurable signals.

def track(name: str = None, phase: Phase = None):

    def decorator(func):

        tags = [f”phase:{phase.value}”] if phase else []

        @opik.track(name=name, tags=tags)

        def wrapper(*args, **kwargs):

            return func(*args, **kwargs)

        return wrapper

    return decorator

If I ever switch observability tools, business code doesn’t change.

What I’m Exploring Next

Next steps include:

1. Adding durable workflow orchestration (DBOS or Prefect)

2. Implementing systematic evaluation for retrieval quality and faithfulness

3. Exploring more advanced retrieval patterns

Each will be added deliberately, one constraint at a time.

Closing Thoughts

Moving from keyword search to semantic and multimodal understanding is a massive leap in how we solve problems. While this technology introduces an ambiguity that contrasts with the deterministic systems I’ve built before, the incredible advantages and sheer problem-solving power it offers make the challenge truly exciting.

Building RAG this way slowed me down, deliberately.

What I have now is a system I can inspect, rerun, and reason about when something goes wrong. For me, that’s a better foundation than a faster demo.

I’m still learning RAG. But I’m learning it with the same instincts that shaped the rest of my career: make systems observable, design for failure, and own the control flow before adding abstraction.

Code: https://github.com/CalvHobbes/rag-101

Inspired by: From 100+ AI Tools to 4: My Production Stack by Paul Iusztin

See you next time.

Priya

What’s your opinion? Do you agree, disagree, or is there something I missed?

Leave a comment

Enjoyed the article? The most sincere compliment is to share our work.

Share

Go Deeper

Go from zero to production-grade AI agents with the Agentic AI Engineering self-paced course. Built in partnership with Towards AI.

Across 34 lessons (articles, videos, and a lot of code), you’ll design, build, evaluate, and deploy production-grade AI agents end to end. By the final lesson, you’ll have built a multi-agent system and a capstone project where you apply everything you’ve learned on your own.

Three portfolio projects and a certificate to showcase in interviews. Plus a Discord community where you have direct access to other industry experts and me.

Rated 4.9/5 ⭐️ by 290+ early students — “Every AI Engineer needs a course like this.”

Learn more

Not ready to commit? We also prepared a free 6-day email course to reveal the 6 critical mistakes that silently destroy agentic systems. Get the free email course.

Images

If not otherwise stated, all images are created by the author.

🧐 Everyone says you need AI evals. Few explain how to actually build them and answer questions such as…

How do we avoid creating evals that waste our time and resources? How do we build datasets and design evaluators that matter? How do we adapt them for RAG? ...and most importantly, how do we stop “vibe checking” and leverage evals to actually track and optimize our app?

This 7-article series breaks it all down from first principles:

1. Integrating AI Evals Into Your AI App

2. Build an AI Evals Dataset from Scratch

3. Generate Synthetic Datasets for AI Evals

4. How to Design Evaluators

5. How to Evaluate the Evaluator ← You are here

6. RAG Evaluation: The Only 6 Metrics You Need

7. Lessons from 6 Months of Evals on a Production AI Companion

By the end, you’ll know how to integrate AI evals that actually track and improve the performance of your AI product. No vibe checking required!

Let’s get started.

How to Evaluate the Evaluator

Your evaluators are running. They produce Pass or Fail verdicts on your agent’s outputs. But one open question remains: how do you know if those verdicts are correct?

While building Brown, a writer agent I developed with the Towards AI team for our Agentic AI Engineering course, I set up an LLM judge to verify generated articles. I wanted to check the expected structure, idea flow, and content against a golden dataset. I ran it on a batch of traces, and the scores seemed reasonable. Then I manually compared the traces against the judge’s verdicts, only to realize it was fixating on the wrong things.

It scored 0 when an article used bullet points instead of H3 headers, which was perfectly acceptable for that section. It scored 0 when the agent used a different transition phrase than the few-shot examples, penalizing creativity when we wanted flexibility. Furthermore, it scored 1 when paragraphs did not flow smoothly into each other, completely overlooking a real quality issue we cared about.

We had to iterate on the judge until it reflected what we actually valued. Anthropic reports a similar pattern, seeing eval scores jump from 42% to 95% after fixing grading bugs and ambiguous task specifications [3]. The agent was fine all along, but the evaluator was broken. That experience crystallized something for me: eval metrics you cannot trust are worse than no metrics at all.

Unvalidated evals create false confidence. You see green dashboards, assume quality is fine, and stop looking. You push broken outputs because the numbers said they were good, and you hear about problems from frustrated users instead of your test suite. Worst of all, you cannot tell which evaluations are wrong, as the 10-20% of incorrect signals hide silently and contaminate every decision built on those scores.

Your evaluator is another AI model that makes binary predictions, so it needs a test set, metrics, and mapped failure modes like any other model.

Also, LLM judges are inherently non-deterministic, meaning they hallucinate, carry biases, and drift. Alignment with human evaluators varies widely by task, with some teams achieving high agreement after careful iteration, while others struggle to break 70% on subjective criteria. The gap between your judge and reality could mean hundreds of bad signals across a thousand evaluations, which you will not know without validation [2].

Here is what you will learn to solve this problem:

• Partitioning your labeled data to prevent data leakage.

• Quantifying agreement using standard classification metrics.

• Systematically closing the gap between your judge and domain experts.

• Dealing with the randomness of LLMs.

To start this process, we first need to structure our dataset correctly.

But before digging into the article, a quick word from our sponsor, Opik. ↓

Opik: Automated Agent Optimization Using Your Data (Sponsored)

This AI Evals & Observability series is brought to you by Opik, the LLMOps open-source platform used by Uber, Netflix, Etsy, and more.

We use Opik daily across our courses and AI products. Not just for observability, but now to automatically optimize our agents’ prompts using the same datasets and metrics we already have in the platform.

You are learning how to build diverse synthetic datasets to evaluate your AI app. But once you have those datasets and metrics, why stop at measuring quality? Opik’s agent optimizer closes the loop. It uses your eval dataset to automatically improve your prompts. Here is why we love it:

• Same datasets, zero extra setup — Opik’s optimizer reuses the exact datasets, metrics, and tracing you already have. Quick start guide.

• Six optimization algorithms — Choose from strategies like HRPO (our favorite), which performs root-cause analysis on failures and proposes targeted fixes, or evolutionary optimization to explore diverse prompt structures. See all algorithms.

• No-code Optimization Studio — For quick iterations, run optimization directly from the Optimization Studio UI. Start from your prompt, pick your dataset, choose an algorithm, and watch Opik test prompt variations against your metrics in real time.

Opik is fully open source and integrates with OpenAI, Anthropic, Gemini, and 100+ providers. Start optimizing your agents for free:

Automated agent optimization guide

↓ Now, let’s move back to the article.

Structuring Your Data for Validation

You already have your ground truth. As explained in Article 2 and Article 3 of the series, your domain expert labeled each trace as Pass or Fail with a critique. Those labels are the reference standard your automated judge must match. If the task is highly subjective, consider having multiple people label the same examples to discover the agreement ceiling, but for most teams, the single expert is sufficient.

Now you need to partition that labeled data correctly. Why? Because you cannot build and validate on the same examples, as that is like grading your own exam. You must calculate the error on unseen data only to make sure you are not getting biased results, so split your dataset into three sets: train, dev and test [2].

The train set takes 60% of the data, representing the examples your evaluator learns from. They go into the few-shot prompt, inform the rubric, and set the standard for what Pass and Fail look like. The dev set takes 20% of the data, acting as your iteration sandbox. Run the judge here, check where it disagrees with the expert, adjust the prompt, and repeat to refine the system. Finally, the test set takes the remaining 20% and must be kept locked until you are done iterating. You use it only at the end when the LLM judge is aligned with the expert on the dev set. This gives you an unbiased final score on data that the evaluator has never seen.

The 60/20/20 split is a good starting point, but as your data grows and you don’t want to overload your few-shot-examples (they grow your context window), you can start moving more data to the dev and test splits.

In practice, 100 labeled examples mean 60 powering the prompt, 20 for tuning, and 20 for the final honest check. Aim for at least 100 labeled examples to get stable metrics. Below 50, your numbers become too noisy to act on. Watch out for class imbalance. If 90% of your traces are Pass and only 10% are Fail, you need a way to balance the classes, either by synthetically increasing your negative class or removing samples from your positive class, until a balance is achieved.

With data properly structured, let us quantify how well your judge actually agrees with the expert.

Measuring Alignment With Human Judgment

Your judge outputs Pass or Fail for each trace, which means you are building a binary classifier. You are using LLMs instead of other models, but ultimately, it’s still just a classifier.

Thus, you need to quantify the performance of the LLM Judge against the golden dataset we just split in the previous section. Standard classification metrics give you this visibility.

The confusion matrix shows four possible outcomes. True Positive (TP) means both judge and expert say Pass, agreeing the output is good. True Negative (TN) means both say Fail, agreeing the output is bad. False Positive (FP) means the judge says Pass, but the expert says Fail, letting a bad output through. False Negative (FN) means the judge says Fail, but the expert says Pass, meaning the judge was overly harsh.

Combining TP, TN, FP, and FN yields three fundamental metrics:

1. Accuracy is the overall agreement rate, calculated as (TP + TN) / total. If the judge matches the expert on 170 out of 200 traces, that is 85% accuracy. This is useful when Pass and Fail are roughly balanced, but it is highly misleading when they are not.

2. Precision measures how trustworthy the Pass verdicts are, representing the fraction of judge-approved traces that the expert also labeled Pass. You calculate it as TP / (TP + FP). If the judge approves 50 articles and the expert disagrees on 8, precision is 42 / 50 = 84%, meaning when the judge says the output is good, you can generally believe it.

3. Recall measures how many actual Passes the judge finds out of all the traces the expert labeled Pass. You calculate it as TP / (TP + FN). If 60 articles are genuinely good but the judge only catches 48, recall is 48 / 60 = 80%, meaning the judge finds most quality output but still misses some.

Ultimately, we have the F1 score as an aggregate metric that provides a balanced view as the harmonic mean of precision and recall, calculated as 2 × (Precision × Recall) / (Precision + Recall). Use this when both false positives and false negatives matter equally. The right F1 target depends on the metric. With Brown, we accepted around 60% for subjective metrics like style, but demanded over 90% for objective ones like article structure. As a general rule, aim for an F1 above 0.70.

These metrics seem simple enough. But there is a common trap most teams fall into when their datasets are not balanced.

When High Scores Hide Real Failures

We can best understand this phenomenon by looking at a few examples.

For example, let’s assume Brown generates 80 articles. 70 are correct, and 10 are broken. Your judge labels every single one as Pass. Accuracy sits at 70 / 80 = 87.5%, which looks reasonable, but it never caught a single failure, making it completely useless.

Let us look at another example in more depth. Out of 80 generated articles, 60 are genuinely well-structured, while 20 have real problems like missing sections or disconnected paragraphs. The judge correctly approves 55 of the good ones and wrongly rejects 5. Of the 20 broken articles, it catches only 4 and lets 16 slip through. That gives us TP=55, FN=5, FP=16, TN=4.

Overall accuracy reads (55 + 4) / 80 = 73.75%, which looks reasonable. But Fail-class recall is just TN / (FP + TN) = 4 / (16 + 4) = 20%, meaning the judge misses 80% of structural failures. The lesson here is to always check precision and recall on the minority class. If those numbers are low, enrich your few-shot prompts with more failure examples, focusing particularly on the subtle cases where individual paragraphs look fine but do not connect fluidly [2].

Now that you know what to measure and what to watch out for, let us walk through the process of systematically improving your judge.

Closing the Gap Between Judge and Expert

This is the core workflow for making your judge reliable. Start with 10-20 few-shot examples from the train set to build your initial judge, and run it against the dev set while leaving the test set untouched. Compute precision, recall, and F1, then identify every disagreement where the judge and expert diverge. Expand your few-shot examples by incorporating those disagreements into the prompt when they reveal real patterns, re-run, and re-measure until the dev set alignment hits your target threshold.

Remember that your few-shot examples translate to input tokens, which translate to extra costs. Thus, ideally, you want to keep your few-shot examples as minimal, yet diverse, as possible, while maximizing performance on your dev and test splits.

Lock the test set for the final check. Only run the judge on the test set after you stop iterating on the dev set, giving you an uncontaminated measurement of real performance.

Expect at least 3 rounds of iteration. If you are still far below target after 10 iterations, the task may require human judgment that no prompt can replicate. Start by hand, as manual prompt refinement teaches you where your judge’s reasoning diverges from the expert’s. Carefully studying each disagreement is the most informative signal you have, and once your labeled dataset is large and high-quality enough, you can explore automated prompt optimization tools.

Read the LLM Judge critiques instead of just looking at metrics, as critiques tell you whether the judge was wrong or the expert missed something. As highlighted by Anthropic, you shouldn’t take eval scores at face value until someone digs into the details and reads the critiques of the judge [3].

Once your judge passes validation, put it to work for regression testing, optimization, and production monitoring as explained in Article 1.

What if the agreement stays low? If after 10 rounds your agreement is still low, here is what to look out for. Your few-shot examples might be too narrow, so as you keep sampling more production traces using your observability platform (e.g., Opik) revisit error analysis, as exlained in Article 2, to find the specific patterns where the judge fails and add those to the few-shot-examples. With Brown, our initial examples were too uniform, and adding subtle structural failures immediately improved alignment.

The rubric might lack specificity, as asking if the article is well-written invites interpretation, while asking if it contains well-defined paragraphs, transitions, and metaphors leaves less room for ambiguity. Sharpen the criteria.

Also, in case the task itself is too subjective, consider accepting a lower F1 score. For example, with Brown, style adherence was inherently subjective, so we accepted a lower F1 there while holding structure to >90%. The idea is to adapt your acceptance threshold based on the nature of each business metric you are tracking.

Even with strong agreement, there is one more challenge. Both your judge and your agent introduce randomness into every run. Let’s see how we can fix that.

Dealing With Non-Determinism

Randomness comes from two directions: as the judge produces different scores on the same input, and the agent itself takes different paths each run. You need to address both to build a stable evaluation pipeline.

The easiest and most powerful way to win is to scale the dataset, as larger datasets smooth out noise. Aim for enough examples in each class that a few misclassifications do not swing your metrics wildly. A good starting point is a minimum of 50 samples per class.

Also, another easy win (but not necessarily cheap) is to pick the strongest available model, using a frontier model like the latest versions of Claude Opus or Gemini Pro, because the judge should be at least as capable as the system it evaluates [2]. Require reasoning before the verdict by structuring the prompt with Chain of Thought (CoT) so the judge walks through each criterion first before delivering Pass or Fail. This step-by-step analysis produces more consistent scores and better alignment with human judgment [1].

Let the judge abstain by giving it an “Unknown” option when it lacks enough information to decide, because forcing a binary Pass/Fail on ambiguous cases generates false positives you cannot distinguish from real ones [3].

To further stabilize the judge, you can compute a significance threshold by running the evaluation 3-5 times and computing the variance between the runs. With Brown, this was essential because writing is subjective, and running the evaluator 5 times told us the real error threshold. A 3% metric shift across runs was noise, but 10% meant something actually changed. Without this, you are chasing random fluctuations.

On the agent side, treat it as a black box and evaluate the destination, not the route, as agents can reach the same outcome through different strategies. Brown might outline first, then write or draft everything, then restructure, but both can produce a strong article. Score the final output against your quality criteria, not the intermediate steps [3].

For the agent, measure reliability across multiple runs using pass@k and pass^k. pass@k tracks whether at least one out of k attempts succeeds, while pass^k tracks whether all k attempts succeed. These two metrics tell opposite stories as k grows: pass@k climbs toward 100% while pass^k dropping sharply, revealing how consistent your agent really is. [3].

You now have the complete toolkit for evaluating your evaluator.

Demo

To fully grasp the end-to-end workflow for building AI Evals, I recommend rewatching our demo using AlignEval, an open-source tool created by Eugene Yan. It provides a streamlined interface for the exact workflow this article teaches: look at your data, label it, evaluate outputs, and optimize your evaluators:

The tool is open source and available at aligneval.com, with the source code on GitHub (eugeneyan/align-app). You can try it for free with your own data or use the prompt below to quickly generate a CSV similar to the one from the demo:

I want you to generate a CSV file with the following characteristics:
"""
* The CSV file must include the following columns:
   * id: Unique identifier for each row
   * input: Context used to generate output
   * output: Generated text to be evaluated
   * label: Ground truth (values optional but counts towards XP)
   * explanation: A one-sentence explanation on why we labeled the row as 0 (PASS) or 1 (FAIL)
* 🚨 The label column only accepts binary labels, either 0 or 1.
   * 0: Output PASSES your evaluation
   * 1: Output FAILS your evaluation
"""
that contains 100 rows

The goal of the CSV file is to implement a dataset to build an LLM Judge evaluator. 

We want to create some mock, synthetic data to conceptually show how labeling, evaluating and optimizing the LLM judge would look like, based on this tool: https://aligneval.com/

Let's say that we collected data from a vertical assistant agent specialized in answering work emails and Slack messages. Thus, create 100 scenarios based on these dimensions:
* feature: email/slack
* scenario: executive, manager, colleague, spam email, phishing email, friend (as an exception)
* label: success/failure of properly answering the message

Where the input is a single email or Slack message or an email or Slack thread, but the output will ALWAYS be just the generated reply, whether it's email or Slack.

Make the labels a 50%/50% split between passes and fails.

Also, note that NO REPLY is an expected behavior for SPAM and phishing emails. Also, for non-essential emails or toxic or slack messages.

We used Claude Opus 4.6 within the Claude app to generate it.

Next Steps

An evaluator only earns trust when it matches expert judgment. The workflow is straightforward: measure where your judge disagrees with the expert, fix those gaps, and confirm on data the judge has never seen. Only when the judge aligns with the expert on the test set can you rely on your eval metrics.

The error analysis workflow and iterative labeling were only the tip of the iceberg. Now you see the full picture of how to build, validate, and maintain evaluators.

Next up is a specialized article focused on evaluating Retrieval-Augmented Generation (RAG) systems.

Also, remember that this article is part of a 7-piece series on AI Evals & Observability. Here’s what’s ahead:

1. Integrating AI Evals Into Your AI App

2. Build an AI Evals Dataset from Scratch

3. Generate Synthetic Datasets for AI Evals

4. How to Design Evaluators

5. How to Evaluate the Evaluator ← You just finished this one

6. RAG Evaluation: The Only 6 Metrics You Need

7. Lessons from 6 Months of Evals on a Production AI Companion

See you next Tuesday.

Paul Iusztin

What’s your opinion? Do you agree, disagree, or is there something I missed?

Leave a comment

Enjoyed the article? The most sincere compliment is to share our work.

Share

Go Deeper

Go from zero to production-grade AI agents with the Agentic AI Engineering self-paced course. Built in partnership with Towards AI.

Across 34 lessons (articles, videos, and a lot of code), you’ll design, build, evaluate, and deploy production-grade AI agents end to end. By the final lesson, you’ll have built a multi-agent system and a capstone project where you apply everything you've learned on your own.

Three portfolio projects and a certificate to showcase in interviews. Plus a Discord community where you have direct access to other industry experts and me.

Rated 4.9/5 ⭐️ by 290+ early students — “Every AI Engineer needs a course like this.”

Learn more

Not ready to commit? We also prepared a free 6-day email course to reveal the 6 critical mistakes that silently destroy agentic systems. Get the free email course.

Thanks again to Opik for sponsoring the series and keeping it free!

If you want to monitor, evaluate and optimize your AI workflows and agents:

Try Opik for free

References

1. Arize AI. (n.d.). LLM as a Judge: Primer and Pre-Built Evaluators. Arize. https://arize.com/llm-as-a-judge/

2. Husain, H. (n.d.). Using LLM-as-a-Judge for Evaluation. hamel.dev. https://hamel.dev/blog/posts/llm-judge/

3. Anthropic. (n.d.). Demystifying Evals for AI Agents. Anthropic Engineering Blog. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

Images

If not otherwise stated, all images are created by the author.

He’s skipping the textbook definitions today to focus on the architectural trade-offs and real-world logic behind his most recent builds.

Enough chitchat. Let’s get into it 👀 ↓

When Single-Agent Systems Fall Apart

You know the moment. You built a perfectly capable AI agent that writes code, answers questions, and searches through your docs. It works great. Then you ask it to review code for security issues and synthesize three different research papers. It returns something that’s half right and half wrong, delivered with full confidence.

I used to think this was a model problem. Better prompts, bigger context window, maybe switch to the latest Sonnet release. Wrong. The problem is architectural, and no amount of prompt engineering fixes it.

A single agent with 40+ tools, a 2,000-word prompt over five different domains, and retrieval tuned for one job at a time collapses. Context windows get bloated. Tool selection becomes a mess. Quality tanks.

This happened to me with Screech, a personal agent I built for my side projects. It started simply, basically a smarter search over my notes. Then I kept adding: code generation, documentation, code reviews, security audits, and research synthesis. The single-agent approach worked beautifully until it very suddenly didn’t.

The stack is not exotic. It’s VoltAgent for runtime and workflows, SurrealDB as the “one DB to store everything” experiment, and Claude as the default model tier.

And yes, the agent is named after Screech from the Saved by the Bell TV series. Also, my childhood nickname.

I didn’t invent this in a vacuum. Graphiti shaped how I think about knowledge that changes over time. VoltAgent gave me workflow primitives I didn’t have to build. Paul Iusztin’s AI Agents Foundations convinced me to stop forcing PDFs through OCR and treat them as images. Agentic Playbooks showed me that auditable agent decisions are a performance win, not only a governance check.

So, here’s the architecture, decisions, and stuff I’d do differently. For legal reasons, that’s “informational only,” not “you should do this.”

Before we continue, a quick word from the Decoding AI team. ↓

Go Deeper: Your Path to Agentic AI for Production

The Agentic AI Engineering course, built in partnership with Towards AI, walks you through building exactly this kind of multi-agent architecture across 34 lessons.

Across 34 lessons (articles, videos, and a lot of code), you’ll design, build, evaluate, and deploy production-grade AI agents end to end. By the final lesson, you’ll have built a multi-agent system and a capstone project where you apply everything you’ve learned on your own.

Three portfolio projects and a certificate to showcase in interviews. Plus a Discord community where you have direct access to other industry experts and me.

Rated 4.9/5 ⭐️ by 290+ early students — ”Every AI Engineer needs a course like this” and ”an excellent bridge from experimental LLM projects to real-world AI engineering.”

Start learning today

↓ Now, back to the article.

If You’re Not Building Agents

You can stop here and still get the gist. The problem: one AI agent that should do everything (search your notes, write code, review for security, summarize research) does all of it poorly. One large prompt and many tools. It confuses tasks, wastes tokens, and returns confident nonsense when goals conflict, e.g., security paranoia versus “ship it” code gen.

The solution: one conductor agent that handles simple work itself and a pool of specialists it calls when the task needs depth. The conductor stays cheap and fast for most requests. Specialists run only when needed. You need routing (who handles what), hybrid retrieval (not only vector search), and one store for documents, relationships, and chat (here, SurrealDB). The rest of this article is for people who want to see the wiring.

Multi-Agent: When It’s Worth the Complexity

The maintenance overhead is real. So let me be clear about when this makes sense.

I’d only do it when there are 3+ domains that actively conflict (dev, research, security is a classic triangle), when I care about cost per request (not “cost later”, cost now), and when I need failures to be contained so that one specialist can be dumb without contaminating the whole system.

Stay single-agent when tasks are similar, the tool count is under about 15, you do not need different model tiers, and simplicity beats per-task quality.

Single-agent favors simplicity. Multi-agent favors quality per task and adds orchestration. Pick your poison.

The Three-Layer Architecture

I think of Screech as an orchestra. One conductor who knows the entire score but doesn’t play every instrument. Backed by specialists who are genuinely brilliant at their specific parts.

Layer 1: Orchestration. It does the boring-but-hard parts: understanding intent, pulling context, and deciding whether this is “handle it now” or “call a specialist”. Three meta-tools carry most of the orchestration weight: discover subagents, invoke one subagent, and fan out to multiple subagents. A task router (Claude Haiku) classifies complexity before any expensive model runs. The runtime, memory management, workflow engine (with suspend/resume), and MCP server integration come from VoltAgent. I didn’t build any of that infrastructure, I plugged in.

Layer 2: Specialists.128 subagents in 10 categories: core development, language specialists (TypeScript, Python, Rust, plus 19 more), testing & quality, meta-orchestration. More on why that number and why these categories in a bit.

Layer 3: Knowledge. Hybrid retrieval combining vector search + knowledge graph traversal + keyword matching, all backed by SurrealDB. Plus a temporal layer (Graphiti-style) so the system knows when it learned something, not only what.

Here’s the key decision: Screech is a full agent with its own tools and retrieval, not a dumb router. That is the decision that matters. It handles 60–70% of requests directly. Subagents only kick in when you need deep specialization. That keeps latency and cost sane for the common case.

If you’ve used Claude Code, this pattern will feel familiar, but there is a key difference: Claude Code is one agent plus injected context (CLAUDE.md, conventions, slash commands). When you give it a task, the same agent handles everything, and it just gets extra context injected from your skill files. It’s the “enhanced single-agent” end of the spectrum: one brain, augmented with domain knowledge. Screech pushes further along that spectrum. Instead of injecting domain knowledge into one agent’s prompt, each specialist is its own agent with a dedicated system prompt, model tier, and tool set. The orchestrator doesn’t just get “React knowledge” injected — it delegates to an react-specialist agent that was born and bred to think in components, hooks, and JSX. The difference matters when domains actively conflict: a security auditor’s “assume everything is dangerous” mindset would poison a code generator’s “keep it simple” prompt if they shared the same context. Separate agents, separate prompts, no cross-contamination. Think of it as: Claude Code = one chef who reads different recipe books depending on the dish. Screech = a head chef who delegates to a pastry specialist, a sushi chef, and a grill master: each with their own kitchen and knives.

The diagram below shows how these layers connect. Here’s the flow from top to bottom:

User Interfaces layer exposes three entry points: a web UI (React), an MCP server (for IDE integration like Cursor), and a CLI terminal. All hit the same orchestration layer.

Screech Web UI is a React-based interface for the Screech personal knowledge agent. It connects to the Screech backend and provides five main views: Sources (ingest and manage documents, view chunk/entity/pattern/insight stats), Notes, Chat (conversation with the agent), Search (query over your knowledge), and Knowledge Graph (browse entities and relationships). It also shows connection status and supports running synthesis. Its main use is to browse, chat, search, and explore your knowledge graph in one place.

Orchestration Layer has as a master agent the Screech agent (Claude Sonnet 4), which sits at the center, handling 60–70% of requests directly. Three supporting components surround it: the Task Router (Haiku, $0.0025/classification), the Event Bus (in-process pub/sub), and Persistent Memory (conversation history, user context).

The 128 Subagents are arranged by category: Core Dev (11), Language Specialists (22), DevOps (15), Testing & Quality (13), Domain-Specific (27), Business (12), Research (6), Dev Experience (12), and Meta-Orchestration (10). The orchestrator delegates to these when deep specialization is needed.

Hybrid Retrieval sits between the agents and the database: 0.6 vector + 0.2 graph + 0.2 keyword, merging three signals before final relevance scoring.

SurrealDB acts as the persistence layer, split into three logical stores in one database: the Vector Store (MTREE index, 3072-dim embeddings, cosine similarity), the Knowledge Graph (entities, relationships, BFS traversal), and the Temporal Graph (Graphiti-inspired episodes, facts, time-range queries).

At a code level, Screech is just an Agent instance with four things wired in: a model, a hybrid retriever, a tool set, and persistent memory. This “agent factory” is the single place where the orchestration decisions become concrete.

// The Screech agent factory
const agent = new Agent({
  name: "Screech",
  purpose: "Unified personal agent for side projects, knowledge synthesis, " +
           "development, documentation, and orchestration of specialist subagents.",
  model: anthropic("claude-sonnet-4-20250514"),
  retriever,  // Hybrid RAG (vector + graph + keyword)
  tools: screechTools,  // Deduplicated from 3 domains
  memory,  // LibSQL persistent memory
});

11 Tables, One Database: The SurrealDB Model

One of the strongest arguments for SurrealDB: documents, embeddings, knowledge graph, temporal events, and conversation memory in 11 tables, one database. No Postgres + Neo4j + Redis dance.

Documents and Chunks

Ingest a document. Create a document record (metadata, content hash for dedup). Then split it into chunk records. Each chunk gets a 3072-dim embedding (OpenAI text-embedding-3-large). SurrealDB’s MTREE index does cosine similarity. MTREE is a tree index for high-dimensional vectors (same idea as pgvector’s HNSW/IVFFlat). It lets SurrealDB find the nearest embeddings without brute-force scanning every row. Chunks are multimodal. They store image_data (base64) and mime_type alongside text. This comes straight from Paul Iusztin’s insight: stop forcing PDFs through OCR. Treat them as images.

// Multimodal chunk structure
interface Chunk {
  document_id: string;  // Parent document link
  content: string;      // Text or image description
  embedding: number[];  // 3072-dim vector (MTREE indexed)
  mime_type?: string;   // "text/plain", "image/png", "application/pdf"
  image_data?: string;  // Base64 for vision-processed pages
  page_number?: number; // PDF page tracking
}

Entities and the Graph (Your Ontology)

Here’s where Screech diverges from typical RAG: I extract a structured knowledge graph. Claude identifies entities and relationships from each document. SurrealDB’s RELATION type makes this straightforward: entity table relates_to with TYPE RELATION IN entity OUT entity. No separate graph DB.

-- SurrealDB graph relationships (native support)
DEFINE TABLE relates_to SCHEMAFULL TYPE RELATION IN entity OUT entity;

RELATE entity:react->relates_to->entity:nextjs CONTENT {
  relation_type: "EXTENDS",
  confidence: 0.9,
  description: "Next.js extends React with SSR and routing"
};

I picked 11 entity types on purpose. This is the system’s ontology, the vocabulary it uses to classify everything it learns: concept, person, organization, tool, technology, pattern, best_practice, principle, process, document, topic. Each type has its own extraction prompt (e.g., person for roles and affiliations, technology for use cases, and ecosystem). Relationship types include IMPLEMENTS, USES, DEPENDS_ON, PART_OF, EXTENDS, SIMILAR_TO. The ontology is deliberately small; there is enough granularity for useful graph queries without turning into a taxonomy nightmare. Bigger ontologies mean more edge cases and more “is this a tool or a technology?” ambiguity. Eleven types cover 95%+ of what a personal knowledge agent encounters.

Episodes and Facts: Temporal Layer (It’s a Log)

This is the Graphiti-inspired layer that most RAG systems completely skip. Every ingestion creates an episode. Think of episodes as an append-only log of everything the system has ever learned. It is time-stamped and immutable. Episodes link to entities via source_episode_ids. Ingest a PDF, and you get a new episode. Process a paper, and you get a new episode. They do not get updated or overwritten. Old episodes don’t get deleted when new ones arrive; they stay in the timeline with their original timestamps. You can ask “what did I know about X six months ago?” and get a real and accurate answer.

Facts are triples (subject, predicate, object) with a source_episode_id. They capture the structured knowledge extracted alongside each episode. When two facts conflict, e.g., “Bun is experimental” (June) and “Bun is production-ready” (January), the agent can prefer the more recent one.

Why does this matter? Knowledge changes. Without temporal tracking, both facts coexist in your knowledge base with equal weight, and the agent might confidently cite the stale one. Graphiti calls this “bi-temporal awareness”. Tracking both when a fact was true in the world and when the system learned it.

Behind the scenes, temporal queries run entity search (match query terms), then episode retrieval in a time range (filter out chat episodes, keep knowledge episodes), then relevance filtering and linking back to entities. The result is a time-ordered context. The context field returned is prompt-ready: entity descriptions and relationship sentences in plain language.

// Time-aware queries
const recent = await queryTemporalGraph("recent learnings", {
  includeTemporal: true,
  timeRange: { start: oneMonthAgo, end: now },
});
// Returns: episodes + linked entities + relationships, time-ordered

Patterns and Insights: The “So What?” Chain

Beyond storage, Screech runs a synthesis pipeline. Detects patterns across your knowledge base. Generates actionable insights. The pattern table stores detected patterns (workflows, successes, failures, optimizations) with embeddings for searchability. The insight table stores generated insights, each linked back to source patterns with relevance scores.

Documents become chunks, which get embedded. Chunks become entities through extraction. Entities become patterns once the synthesis pipeline starts noticing recurring signals. Patterns become insights (actionable takeaways with provenance). Each stage feeds the next. Every layer is searchable. Vague question goes to vector over chunks. Relationships go to a graph over entities. “What should I do?” goes to insights.

Conversation Memory

user, thread, message tables handle conversation memory. Zep-style user summaries, conversation threading, and message history. Persistent context across sessions, but separate from the knowledge base.

The diagram below shows how data flows through the system. Think of it as five layers stacked on top of each other, each feeding the next:

1. Document layer (top): document and chunk. Raw material comes in here. A PDF becomes a document record; its content gets split into chunk records, each with a 3072-dim embedding. This is the foundation everything else builds on.

2. Temporal layer: episode and community. Every ingestion event creates an episode timestamped to when the system learned it. Episodes link back to chunks (what was ingested) and forward to entities (what was extracted). This is the Graphiti-inspired time dimension—the system knows when it learned something, not just what.

3. Knowledge graph layer: entity, relates_to, and fact. Entities extracted from chunks (concepts, technologies, people) live here, connected by typed relates_to edges. The diamond shape in the diagram represents the relationship table, which is a SurrealDB RELATION type that sits between entity nodes. fact triples (subject, predicate, object) capture the structured knowledge extracted alongside entities.

4. Synthesis layer: pattern and insight. Patterns detected across your knowledge base (recurring workflows, success/failure signals, optimization opportunities) and actionable insights generated from those patterns. Each links back to the entities and episodes that sourced it.

5. Conversation layer (bottom): user, thread, message. Conversation memory, separate from knowledge. Threads reference the user; messages reference threads. The agent can query conversation history independently of the knowledge base.

The arrows in the diagram show the key relationships: chunks link to their parent document. Episodes link to chunks and entities. Entities connect via relates_to. Patterns and insights link back to entities and episodes for provenance. Each layer is independently searchable via the hybrid retrieval pipeline.

When to Use This vs. Alternatives

Unified SurrealDB works when you want graph + vector + relational without three databases, your dataset is moderate (thousands to tens of thousands of docs), and you value dev velocity over ecosystem maturity.

For production SLAs, PostgreSQL plus pgvector is the safer bet. If your graph is only 2–3 hops (like Screech’s BFS), Postgres handles it with recursive CTEs or JOINs, even at millions of rows. Neo4j earns its place when you need deep traversals or heavy graph queries. Graphiti uses Neo4j for that. For my 2-hop, few-thousand-entity case, Postgres + pgvector could do it all. I chose SurrealDB to prototype faster with one schema and one connection. Right call for me. One schema file. One connection. One query language. I would not blindly recommend it for a team with compliance needs.

Subagent System: Factory, Registry, Profiles

Every subagent is a factory (memory?) => Agent. Keeps instantiation lazy (no subagent created until needed) and shared memory (agents in the same workflow see the same conversation history).

// Subagent factory pattern
export type SubagentFactory = (memory?: Memory) => Agent;

// Registry metadata for discovery
export interface SubagentDefinition {
  name: string;
  description: string;
  category: SubagentCategory;  // 10 categories
  tags: string[];
  modelTier?: ModelTier;       // fast | standard | reasoning
  toolProfile?: ToolProfile;   // core | dev | security | full
  capabilities?: SubagentCapabilities;
  factory: SubagentFactory;
}

Three decisions that actually matter:

Model tiers control cost. Not every agent needs Sonnet. Simple formatting → Haiku (~90% cheaper). Security audits → reasoning tier. Default is standard (Sonnet 4). Router can override.

const MODEL_MAP = {
  fast: anthropic("claude-3-5-haiku-20241022"),     // ~10x cheaper
  standard: anthropic("claude-sonnet-4-20250514"),   // Balanced
  reasoning: anthropic("claude-sonnet-4-20250514"),  // Same model, deeper prompts
};

Tool profiles prevent token waste. Research analyst doesn’t need git tools. Code reviewer doesn’t need security scanning tools. Four profiles: core (7 tools): Knowledge/RAG + file ops + workflow discovery, dev (15 tools): core + git + code analysis + testing, security (17 tools): dev + security scanning + dependency audit andfull (18 tools): everything (backwards-compatible default)

Each subagent can add domain tools on top of its profile.

Capability declarations. With 128 agents, “find agents tagged typescript” returns a dozen. The orchestrator needs to know what each agent is good at. Each subagent declares what it can do, expected input, output, and latency tier. Semantic matching sends “TypeScript conditional types” to the agent whose canDo includes “conditional types” and “type system design”, not any agent with TypeScript in the tag. Same language field, different canDo, e.g., typescript-pro vs. react-specialist.

capabilities: {
  canDo: ["type system design", "generics", "conditional types"],
  languages: ["typescript", "javascript"],
  inputSchema: "code snippet or type problem description",
  outputSchema: "typed solution with explanation",
  latencyTier: "medium",
}

Why 10 categories? Flat list worked until about 40 agents. Then discovery got noisy. Categories are a coarse filter: the orchestrator picks a category, then finds the right specialist within it. The split follows prompt conflicts: security paranoia vs. code-gen creativity, test-engineer adversarial vs. technical-writer explanatory. Separate categories, separate prompts.

At the same time, agents in the same category share a tool profile but differ in expertise. All language specialists get the dev tool profile (git, testing, code analysis). All testing-quality agents share dev tools, too, but their prompts are tuned for finding problems, not writing code. Security agents get security tools. Research agents only need core tools (knowledge/RAG). The table below summarizes counts and examples. The diagram repeats it visually.

How Voltagent Fits Into the Picture

VoltAgent is the runtime layer that makes the orchestration practical: it provides workflow primitives, tool execution, memory management, and suspend/resume so the orchestrator and subagents can run as a coordinated system.

The $0.0025 Routing Layer

This is what makes the economics work. Before anything expensive runs, I do a tiny classification call (Haiku) to label the request. Complexity. Domain. Suggested tier. Up to three candidate specialists. The system prompt gives Haiku the full category list. Rules are explicit. “Security” or “vulnerability” always goes to reasoning. Simple question goes too fast. Code gen goes to at least standard.

const classificationSchema = z.object({
  complexity: z.enum(["simple", "moderate", "complex"]),
  domain: z.enum(["lookup", "formatting", "code-generation",
                  "code-review", "architecture", "security",
                  "debugging", "research", "orchestration", "other"]),
  reasoning: z.string(),
  suggestedTier: z.enum(["fast", "standard", "reasoning"]),
  suggestedSubagents: z.array(z.string()).max(3),
});

Step 2: Domain overrides. resolveRoutedTier() takes the complexity-based tier and domain overrides and picks the higher of the two. So a “simple” security question still goes to reasoning. Security that looks simple often is not. The override is a safety net for Haiku’s optimism.

const DOMAIN_TIER_OVERRIDES = {
  security: "reasoning",     // No shortcuts
  architecture: "reasoning", // No shortcuts
  debugging: "reasoning",    // No shortcuts
  "code-review": "standard", // At least standard
};

Step 3: Final routing. Resolved tier + suggested subagents + rationale → orchestrator picks model and specialists.

Fallback. If classification fails (network error, timeout), we default to { complexity: "moderate", tier: "standard" }. Fail to the middle. Not cheapest (might undershoot). Not the most expensive (waste on every failure). Safest with zero information.

Cost: ~$0.0025 per classification (~$0.25/M input tokens on Haiku). Route 1,000 tasks, spend $2.50. If even 30% land on Haiku instead of Sonnet, you save on the order $8-10 per 1,000 tasks. The router pays for itself quickly.

Hybrid Retrieval: Three Signals Beat One

I started with pure vector search. It worked until it didn’t. Three failure modes:

1. Structural queries fell flat. “What tools does the API designer use?” The answer is in the relationship structure. Vector search gave me chunks that mentioned the API designer, not the ones describing its tool config. I needed graph traversal.

2. Exact-match queries got paraphrased away. “What is the error for SQLITE_BUSY?” Embeddings map that into “database locking” neighborhood and miss the chunk with the actual error code. I needed a keyword.

3. Long-document questions needed reasoning, not similarity. “What are the conclusions?” The conclusion section often is not the most similar to the word “conclusions”. The intro restating the thesis can score higher. I needed the model to reason over document structure (e.g., a table of contents), not only similarity.

Instead of trying to make one approach handle everything, I split retrieval into three paths:

The agent picks which tool to call: document_search / get_context → hybrid (vector + graph + keyword, then rerank). search_within_document → same pipeline, one document. answer_from_document_deep → build a section tree from chunks, LLM picks sections, fetch those chunks only. No vectors on path 3.

Why Three Signals Beat One (Paths 1 & 2)

The hybrid pipeline runs three queries in parallel, then merges with configurable weights:

const DEFAULT_WEIGHTS = {
  vectorWeight: 0.6,   // Semantic similarity (primary)
  graphWeight: 0.2,    // Relationship traversal
  keywordWeight: 0.2,  // Exact matching
};

Why these weights? Vector search gets 0.6 because most knowledge base questions are conceptual, like “explain X,” “how does Y work?”; therefore, embeddings handle these well. OpenAI’s text-embedding-3-large (3072 dims) with SurrealDB’s native MTREE index, cosine similarity. The threshold is intentionally low (0.35), but you can tweak that value according to your use case. Similarity scores compress into a narrow range, so 0.35 is more selective than it sounds.

Graph search gets 0.2 because structural queries are the minority but high-value. It works in two stages. First, it finds entities that match the query by name or description, basically a simple text search over the entity table (concepts, technologies, people, organizations). Then it expands outward from those matches using breadth-first search (BFS): for each matched entity, it queries the relates_to edges in SurrealDB to discover neighbors, scoring them at 70% of the parent’s relevance. Configurable traversal depth (default: 2 hops) controls how far the expansion goes. It should be deep enough to find meaningful connections, but shallow enough to avoid pulling in the entire graph.

So: vector search finds a chunk mentioning “React”? Graph search starts at the “React” entity node, walks its edges, and pulls in “hooks”, “server components”, or “Next.js”, without needing those terms in the original query. Need the path between two concepts? A separate BFS finds the shortest connection: React → EXTENDS → JavaScript → USES → V8, each hop following typed relationships (IMPLEMENTS, USES, DEPENDS_ON, EXTENDS, SIMILAR_TO). This is the signal that vector search fundamentally cannot provide, because relationships are structural, not semantic.

Keyword search gets 0.2 because sometimes you just need to find the exact string. Ask a pure vector system “what version of React does project X use?” and it’ll confidently return chunks about React 17, React 18.2, and React 19, all because to an embedding model, they’re all basically “React with a number.” Helpful if you’re writing an essay. Useless if you need the actual version pinned in your package.json. Keyword search is the boring friend who actually reads the label. Full-text matching with term coverage scoring. No AI magic, just string comparison. And for error codes, version numbers, and config keys, that’s exactly what you want.

After merging, results go through reranking, and this is where the quality jump happens. The weighted merge gets you close, but reranking catches cases where a high-scoring vector result is semantically related but does not actually answer the question.

The reranker supports three methods, selectable per query:

• Embedding reranking (fast, cheap): recalculates cosine similarity between the query embedding and each result’s embedding, then blends it 50/50 with the original merge score. This catches results that scored well on the graph or keyword but are semantically distant from the actual query. Fast because it’s just math. You don’t need an LLM call.

• LLM reranking (slower, more accurate): sends the query + top 20 candidate passages to Claude Sonnet 4, which scores each on a 0–1 relevance scale. The LLM understands intent, and it knows that “how do I fix CORS errors?” is asking for a solution, not a definition. Sits behind an LRU cache (128 entries, 5-minute TTL) to avoid redundant calls for similar queries.

• Hybrid reranking (two-pass): embedding reranking first to narrow the candidate set, then LLM reranking on the survivors. Best quality, highest latency.

On top of any reranking method, there’s an optional diversity-aware mode using MMR (Maximal Marginal Relevance). It iteratively selects results that maximize relevance while penalizing similarity to already-selected results, so it prevents returning five chunks from the same paragraph. Plus a source-type preference layer that weights chunks, entities, patterns, and insights differently depending on the query type.

Why Reasoning Beats Similarity for Long Documents (Path 3)

This is the insight that took me the longest to internalize. For a 300-page PDF, when someone asks “what are the conclusions?” the location of the answer is a function of document structure, not content similarity. A chunk from the introduction that restates the thesis will often score higher on cosine similarity to “conclusions” than the actual conclusion section. More embedding dimensions won’t fix this. Better chunking strategies help, but don’t solve it.

Path 3 skips vector search entirely. The pipeline has three steps:

Step 1: Build the section tree. Take the document’s chunks (already stored from ingestion) and group them into sections. If chunks have page numbers (PDFs), group by page. Otherwise, group into fixed-size windows (default: 4 chunks per section). Each section node gets an ID, a title (”Page 12” or “Section 5”), and a short summary (first ~220 characters of the first chunk). The result is a flat list of TreeNode objects, essentially a reconstructed table of contents.

interface TreeNode {
  node_id: string;        // "s0", "s1", "s2"...
  title: string;          // "Page 12" or "Section 5"
  summary: string;        // First ~220 chars of first chunk
  startChunkIndex: number;
  endChunkIndex: number;
  pageRange?: string;     // "pp. 12–14"
}

Step 2: LLM selects relevant sections. The tree outline (node IDs + titles + summaries) is sent to Claude in a single prompt. The key instruction: use reasoning, not keyword matching. The prompt explicitly tells the LLM to think structurally, e.g., “conclusions are usually in the final section”, “see Appendix G means look for an appendix section.” The LLM returns a JSON object with its reasoning and a list of selected node IDs.

// What the LLM sees (abbreviated)
// - s0: Page 1 — "Chapter 1: Introduction. This paper presents..."
// - s1: Page 2 — "Related work in retrieval-augmented generation..."
// - ...
// - s14: Page 28 — "7. Conclusions and Future Work. We have shown..."

// What the LLM returns
{
  "thinking": "Conclusions are in the final sections. s14 title mentions Conclusions.",
  "node_list": ["s14"]
}

Step 3: Fetch and return. Map the selected node IDs back to chunk index ranges, fetch those chunks, and concatenate their content. That’s your retrieval context. No embedding comparison anywhere in the pipeline.

The fallback is important: if the LLM returns invalid JSON or no valid node IDs, the system defaults to the first 2–3 sections. Better to return something than nothing, and introductory sections are a reasonable default for most questions.

The design is directly inspired by PageIndex‘s thesis: similarity ≠ relevance, and reasoning over document structure often beats embedding search for professional long-form content. It won’t help for vague conceptual questions; that’s what path 1 is for. But for “where in this document does X live?” or “what does chapter 7 say about Y?”, it’s dramatically better because the LLM can reason about document organization the way a human reader would: by scanning the table of contents first.

Document-scoped (path 2) simply narrows the same hybrid pipeline to one document via a documentIds filter. Same three signals, same reranker—just scoped.

The temporal knowledge graph is directly inspired by Graphiti, Zep’s framework for building real-time knowledge graphs. Their core insight: knowledge isn’t static. Tracking when facts were learned matters as much as the facts themselves. Perfect for a personal agent that continuously ingests new content.

Every ingested piece creates an “episode”; remember that it’s a timestamped event linking to extracted entities and fact triples (subject-predicate-object). So now you can use time-aware queries: “What technologies have I been reading about this month?” or “How has my understanding of RAG changed?”

Four Production Patterns That Actually Saved Me

These emerged from running this thing in the wild. I’d recommend all four to anyone building multi-agent systems.

1. LLM Resilience with Tiered Timeouts

Every LLM call goes through withLLMResilience():wrapper that adds per-attempt AbortController timeouts, exponential backoff with jitter, retry only for rate limits / 5xx/ network. Never retries 4xx errors (your fault, not theirs). Different timeouts per use case: classification 60s (if it takes that long, fail), synthesis 300s (different budget). I learned this the hard way. One stuck call should not hold everything up.

export const LLM_TIMEOUTS = {
  fast: { timeoutMs: 60_000, maxRetries: 3 },      // Classification, reranking
  standard: { timeoutMs: 120_000, maxRetries: 2 }, // Agent generation
  long: { timeoutMs: 300_000, maxRetries: 1 },     // Synthesis, deep analysis
};

// Usage
const result = await withLLMResilience(
  (signal) => anthropic.messages.create({ ... }, { signal }),
  { ...LLM_TIMEOUTS.fast, label: "task-classification" }
);

Key insight: different operations need different timeout budgets. Classification call taking 60 seconds? Failed. Synthesis operation taking 60 seconds? Just warming up.

2. Findings Cache for Review Chains

When you run multiple reviewers on the same code (code-reviewer → security-auditor → design-analyst), each reviewer produces findings that downstream reviewers need. Without sharing, every reviewer re-parses the same files, re-discovers the same structure, and wastes tokens on duplicate analysis.

The FindingsCache is a singleton in-memory cache keyed by chain ID. It stores two things: structural analysis (file structure, dependencies, symbols, complexity metrics) produced by the first reviewer, and accumulated findings from every reviewer in the chain—each typed with category, severity, source, and location.

// Each finding is typed and traceable
interface ReviewFinding {
  source: string;     // Which reviewer produced it
  category: "structural" | "quality" | "security" | "design" | "performance";
  severity: "info" | "low" | "medium" | "high" | "critical";
  summary: string;    // Human-readable
  data?: Record<string, unknown>;  // Structured data per reviewer type
  location?: string;  // File/line reference
}

The first reviewer in the chain caches the expensive structural work. The next ones get getChainContextForReviewer(): previous findings + structural cache as a prompt-ready string. Typed findings (source, category, severity, location). TTL 10 min, cap 50 chains. Cuts chain latency by about 40%. The expensive part is parsing and context building, not the LLM. Pattern credit: Agentic Playbooks. Traceable decisions are also a performance win.

// First one caches structure
findingsCache.setStructuralCache(chainId, {
  fileStructure: "src/api/users.ts - 245 lines, 3 exports",
  dependencies: ["express", "zod", "prisma"],
  symbols: ["createUser", "validateInput", "UserSchema"],
  metrics: { cyclomaticComplexity: 12, loc: 245 },
});

// Next get pre-built context
const context = findingsCache.getChainContextForReviewer(chainId, "security-auditor");
// Returns previous findings + cached structure, prompt-ready

What the Workflow Looks Like in Practice

The screenshot below is a real run of a multi-step workflow, showing the chain of specialist calls and the event-style logging that makes the system debuggable.

3. In-Process Event Bus

Agents need to share findings without tight coupling. Solution: a singleton in-memory pub/sub: typed events, well-known topics (VULNERABILITY_FOUND, CODE_REVIEW_COMPLETE, etc.), source agent ID + correlation ID for tracing across a review chain, and a typed payload. Security auditor finds a vulnerability? Publishes to vulnerability_found. Code reviewer subscribes, incorporates the finding.

The implementation is a single AgentEventBus class with no external dependencies.

// Typed event structure
interface AgentEvent<T = unknown> {
  id: string;           // Auto-generated: evt_<timestamp>_<counter>
  topic: string;        // Well-known topic (e.g., "vulnerability_found")
  source: string;       // Publishing agent name
  data: T;              // Typed payload
  timestamp: string;    // ISO timestamp
  correlationId?: string; // Chain/session tracing
}

The key design choice: fire-and-forget delivery. When an agent publishes, subscribers are notified via Promise.allSettled(). Slow or failing subscribers never block the publisher. Handler errors are caught and logged, never thrown.

// Publisher (security-auditor)
await eventBus.publish("vulnerability_found", "security-auditor", {
  severity: "critical",
  type: "sql-injection",
  location: "src/api/users.ts:42",
});

// Subscriber (code-reviewer) registered at startup
eventBus.subscribe("vulnerability_found", async (event) => {
  // Incorporate into review findings
});

Late-joining subscribers can replay event history (last 50 per topic). Events auto-expire via TTL (5 minutes) with periodic cleanup every 100 events. This way, it keeps memory bounded without needing a background timer. Source filtering lets subscribers only receive events from specific agents.

4. Live Evaluation with Sampling

Production traffic gets evaluated by moderation and relevancy scorers at configurable sampling rates. Moderation runs on 20% of requests (cheap). Relevance scoring on 10% (LLM judge, expensive). Both async, non-blocking. Never slow down user-facing response.

What I’d Change: Real Talk

The in-memory event bus doesn’t survive restarts. Fine for a personal side-project agent. Terrible for a production system serving a team. Durable workflow engines like Prefect, Temporal, or DBOS solve this really well with less infrastructure overhead than rolling your own durability with Redis Streams or NATS. Current design optimizes for simplicity, not resilience. I made that trade knowingly.

128 subagents is ridiculous. Pareto wins: ~20 agents do 80%+ of the work. The long tail exists because adding a subagent costs basically nothing (factory + registry entry). I should prune the ones that never get used. Future me will regret not doing that earlier.

SurrealDB as a unified store is elegant but young. Graph + vector + relational in one database? Architecturally clean. But doc gaps cost me time. For strict SLAs, I would use Postgres plus pgvector. For 2 to 3 hop graphs, Postgres is enough. Neo4j, when you need a deep graph. I chose SurrealDB to move fast with one DB. Wouldn’t push it on a team with compliance requirements.

Haiku routing adds ~500ms latency. Noticeable in interactive chat. Negligible in background workflows. For latency-critical paths, consider static routing rules (if the tool is security_scan always use reasoning tier) and only invoke the dynamic router for ambiguous tasks.

Workflow suspend/resume is powerful but adds state complexity. The 70+ workflows support human-in-the-loop via suspend/resume—workflow pauses, waits for human input, and continues. Great for approval flows (expense reports, code reviews). Terrible for state management. Every suspended workflow is a piece of state that can go stale. I’ve had workflows suspended for weeks because I forgot about them. Take that as you will.

The Elephant in the Room: Multi-Agent Is Hard for Everyone

Screech and others: Claude Code with Skills, custom orchestrators, the lot.

Overconfidence. One wrong assumption in step 2 of a 10-step workflow and you get a confidently wrong result. Isolated specialists with focused prompts help, but don’t remove it. I still see invented APIs and wrong architectural assumptions.

More agents do not mean better output. Coordination overhead, conflicting findings, more for the human to reconcile. Findings cache and event bus make communication explicit and traceable, but someone still has to review. Synthesis is an LLM summarizing other LLMs. The chain can be long.

Oversight tax. You spend more time reviewing and redirecting than writing. PR review times go up in high-adoption teams (e.g., plus 91%). Comprehension debt: the more you delegate, the less you understand your codebase. Review becomes rubber-stamping. Screech does not fix that.

Token bloat. Tool schemas, prompts, skills. You can blow past 50k tokens before the agent does useful work. I keep tool profiles tight (7–18 per agent). Complex runs still burn tokens.

Credentials. For a side project, it’s manageable. For production with real APIs and DBs, auth and secrets become a project. A lot of agent efforts reportedly fail to scale there. Not the AI. The plumbing. (My therapist has asked me not to elaborate.)

I’m building Screech knowing these limits. The design mitigates some of it. It does not remove it. Multi-agent amplifies both capability and failure modes. Build guardrails.

Standing on Shoulders

VoltAgent. TypeScript agent framework. Runtime, memory, workflows, MCP, observability. Saved me months.

Graphiti (Zep). Temporal knowledge graph. Episodes, bi-temporal awareness. “Knowledge changes” shaped my RAG thinking.

Decoding AI, AI Agents Foundations (Paul Iusztin). Treat docs as images, not OCR. ReAct, tools, memory.

Agentic Playbooks (Justin Narracott). Traceable, auditable decisions as a performance pattern. Findings cache and review chains owe a lot here.

PageIndex (Vectify AI). Reasoning over document structure instead of pure similarity. Path 3 (tree-search) is inspired by this.

Three Patterns Worth Stealing

You don’t need 128 subagents or a temporal knowledge graph. Here are the three ideas that transfer to any multi-agent system:

1. Route cheap before routing expensive. A $0.0025 classification call that routes 30% of tasks to a model, 90% cheaper? Pays for itself on the first batch. Even without subagents, using a small model to decide whether a task needs your large model is almost always worth it.

2. Not every agent needs every tool. Tool profiles cut token usage, improve tool selection accuracy, and make prompts focused. A research analyst with 7 tools outperforms the same analyst drowning in 18 tools they’ll never use.

3. Hybrid retrieval beats any single method. Vector search handles 70% of queries. Graph traversal and keyword matching cover the other 30% (structural queries, exact-match lookups, relationship questions that embeddings silently botch).

The multi-agent pattern isn’t inherently better. It’s a trade: quality per task versus orchestration complexity. Start with a single capable agent. When quality degrades across diverse tasks, reach for these patterns. The hard part isn’t the agents. It’s the routing, the retrieval, the resilience.

Screech runs on VoltAgent (agent framework), SurrealDB (multi-model database), and Anthropic Claude (LLM). Architectural inspiration from Graphiti, Agentic Playbooks, PageIndex, and the Decoding AI community. Built for personal side-project workloads. Adapt the patterns to your scale.

‘Till next time

Lucian Lature | LinkedIn

What’s your opinion? Do you agree, disagree, or is there something I missed?

Leave a comment

Enjoyed the article? The most sincere compliment is to share our work.

Share

Go Deeper

Go from zero to production-grade AI agents with the Agentic AI Engineering self-paced course. Built in partnership with Towards AI.

Across 34 lessons (articles, videos, and a lot of code), you’ll design, build, evaluate, and deploy production-grade AI agents end to end. By the final lesson, you’ll have built a multi-agent system and a capstone project where you apply everything you’ve learned on your own.

Three portfolio projects and a certificate to showcase in interviews. Plus a Discord community where you have direct access to other industry experts and me.

Rated 4.9/5 ⭐️ by 290+ early students — “Every AI Engineer needs a course like this.”

Learn more

Not ready to commit? We also prepared a free 6-day email course to reveal the 6 critical mistakes that silently destroy agentic systems. Get the free email course.

Images

If not otherwise stated, all images are created by the author.

🧐 Everyone says you need AI evals. Few explain how to actually build them and answer questions such as…

How do we avoid creating evals that waste our time and resources? How do we build datasets and design evaluators that matter? How do we adapt them for RAG? ...and most importantly, how do we stop “vibe checking” and leverage evals to actually track and optimize our app?

This 7-article series breaks it all down from first principles:

1. Integrating AI Evals Into Your AI App

2. Build an AI Evals Dataset from Scratch

3. Generate Synthetic Datasets for AI Evals

4. How to Design Evaluators ← You are here

5. How to Evaluate the Evaluator

6. RAG Evaluation: The Only 6 Metrics You Need

7. Lessons from 6 Months of Evals on a Production AI Companion

By the end, you’ll know how to integrate AI evals that actually track and improve the performance of your AI product. No vibe checking required!

Let’s get started.

How to Design Evaluators

You have a dataset. You’ve manually labeled examples. You’ve fixed the obvious bugs. Now you need evaluators that can run automatically and catch problems before users do.

But here’s what trips up most teams: they build evaluators that check for things nobody cares about, or they use off-the-shelf metrics that sound impressive but don’t match their actual use case.

Three months ago, I spent a weekend building what I thought was a comprehensive evaluation suite for an AI agent that drafted replies to customer support tickets. I had ROUGE scores, BLEU scores, semantic similarity metrics, the works. Everything from the NLP textbook.

Then I ran it on production traces. The evaluators gave perfect scores to replies that were factually wrong, missed the customer’s actual question, and used the wrong tone for frustrated users. Meanwhile, they penalized perfectly good replies for using “different words than the reference answer.”

That’s when I realized: generic metrics optimize for academic benchmarks, not business outcomes. (And no, I’m not saying academic metrics are useless. They’re just solving a different problem than “did this agent do what my users needed?”)

The solution is to design evaluators that match your specific success criteria. Not what worked for someone else’s summarization task. Not what scored well on SQuAD. What actually matters for your users in your use case.

In this article, we will cover:

• The evaluation harness: infrastructure that runs evals end-to-end

• Dataset and metric types: direct scoring vs. pairwise vs. reference-based

• Model evaluation vs. app evaluation (and why benchmarks lie)

• Components of an evaluator: reference examples, metrics, rubrics

• When to use code-based checks vs. LLM judges

• Common mistakes (and how to avoid them)

• Advanced metric designs for multi-turn conversations and agentic workflows

Before digging into the article, a quick word from our sponsor, Opik. ↓

Opik: Open-Source LLMOps Platform (Sponsored)

This AI Evals & Observability series is brought to you by Opik, the LLMOps open-source platform used by Uber, Etsy, Netflix, and more.

We use Opik daily across our courses and AI products. Not just for observability, but to design and run the exact evaluators this article teaches: custom LLM judges, code-based checks, and experiments. All from the same platform.

This article shows you how to design evaluators. Opik gives you the harness to run them at scale. Here is how we use it:

• Custom LLM judges with rubrics — Build the evaluators this article describes: define your criteria, add few-shot examples, and run them across hundreds of traces automatically.

• Run experiments, compare results — Test different prompts, models, or configurations side by side. Opik scores each variant with your evaluators and shows you which one wins.

• Plug evaluators into production — The same LLM judges you design for testing run on live traces too. Set up alarms when scores drop below your threshold so you catch regressions before users do.

Opik is fully open-source and works with custom code or most AI frameworks. You can also use the managed version for free (with 25K spans/month on their generous free tier):

Try Opik for free

↓ Now, let’s move back to the article.

Understanding the Evaluation Harness

You can’t manually run 500 test cases. You need automation.

The infrastructure that runs evals end-to-end is called an evaluation harness (1). It loads your dataset, executes your agent on each test case, captures all the outputs and traces, runs your graders, and aggregates the scores into something you can actually use.

Think of it like pytest for AI apps. Except instead of checking if a function returns the right type, you’re checking if an LLM generated text that accomplishes a business goal.

Here’s what a harness does:

1. Loads tasks from your evaluation dataset

2. Provides instructions and tools to the agent (system prompts, available functions, etc.)

3. Runs tasks (often in parallel across multiple trials because LLM outputs vary)

4. Records everything: inputs, outputs, tool calls, reasoning traces, intermediate states

5. Runs graders on the results (your evaluators)

6. Aggregates scores across trials and tasks

Without a harness, you’re manually running your agent on test cases and eyeballing the output. With a harness, you run 500 test cases overnight and wake up to a report showing exactly which failure categories spiked [1].

The harness is separate from your evaluators. The evaluators decide what “good” means. The harness handles the boring work of running everything at scale and collecting results.

Popular harness options include Opik (what we use), Braintrust, LangSmith, and open-source frameworks like Promptfoo. But honestly, you can build a minimal harness in ~100 lines of Python if you need custom logic [1]. The hard part isn’t the infrastructure - it’s assembling the right context (system prompts, conversation history, retrieved docs, tools) for each task. The key is having one. Don’t manually run evals.

Now let’s talk about what those evaluators actually check.

Dataset and Metric Types: Three Ways to Grade

When designing an evaluator, you need to pick a grading strategy. There are three main approaches, each suited for different situations.

1. Direct Scoring (Pointwise Evaluation)

The evaluator looks at a single output and scores it in isolation. No comparison to anything else.

Example:

• Input: “Refund my order #12345”

• Output: “I’ve processed your refund for order #12345. You’ll see the credit in 3-5 business days.”

• Score: Pass (correctly identified the task, provided timeline, professional tone)

When to use:

• You have clear, absolute quality criteria (was it helpful? was it safe? did it call the right tool?)

• You want to track performance over time on the same dataset

• Your baseline is “good enough” not “better than X”

Metrics:

• Binary pass/fail

• 0-1 scores (where 1 = perfect)

• Classification labels (Helpful/Neutral/Harmful)

2. Pairwise Comparison

The evaluator compares two outputs and picks which one is better.

Example:

• Input: “Refund my order #12345”

• Output A: “Refund processed.”

• Output B: “I’ve processed your refund for order #12345. You’ll see the credit in 3-5 business days.”

• Winner: Output B (more informative, sets expectations)

When to use:

• Comparing two model versions (baseline vs. candidate)

• A/B testing different prompts

• LLMs are better at ranking than absolute scoring

Watch out for biases (2):

• Position bias: LLMs favor the first or last response shown

• Verbosity bias: LLMs prefer longer answers even when they’re not better

• Self-enhancement bias: LLMs favor outputs from themselves over other models

You can mitigate these by randomizing response order and using multiple trials.

3. Reference-Based Evaluation

The evaluator compares the output to a known “gold standard” answer.

Example:

• Input: “What’s the capital of France?”

• Output: “Paris”

• Reference: “Paris”

• Score: Exact match (Pass)

Example 2 (Semantic equivalence):

• Input: “Summarize the refund policy”

• Output: “Customers can return items within 30 days for a full refund if unused.”

• Reference: “Full refunds are available for unused products returned within 30 days of purchase.”

• Score: Pass (different wording, same meaning)

When to use:

• You have ground truth answers (FAQs, knowledge bases, structured tasks)

• Task has a single correct answer or small set of acceptable answers

• You’re testing retrieval accuracy or factual correctness

How to measure:

• Exact match: For structured outputs (dates, product IDs, categorical values)

• Semantic similarity / LLM judges: For natural language, where multiple phrasings are valid (summaries, explanations, instructions)

Common metrics (3):

• Exact match

• ROUGE (recall-oriented, good for summarization)

• BLEU (precision-oriented, originally for translation)

• BERTScore (semantic similarity using embeddings)

• LLM judges (for nuanced semantic equivalence)

The trap: Exact match metrics penalize valid variations. If your reference says “The meeting is on Friday” and your agent says “The meeting is scheduled for this Friday,” exact match fails. This is where semantic similarity metrics (BERTScore) or LLM judges become powerful - they can recognize that different phrasings convey the same outcome.

Model Evaluation vs. App Evaluation (Why Benchmarks Lie)

Here’s a distinction that matters more than people realize:

Model evaluation measures the LLM itself, in isolation, on generic tasks. This is what benchmarks like MMLU, HumanEval, and Chatbot Arena do.

App evaluation measures your entire application (LLM + prompts + tools + retrieval + business logic) on your specific use case.

High MMLU score doesn’t mean it handles your refund policy correctly. Benchmarks test general capability. You need to test your specific use case.

Model Evaluation (Benchmarks)

Tests: “Can this LLM answer random trivia, write code snippets, or score high on standardized tests?”

Useful for:

• Comparing foundation models across the board

• Understanding general capabilities

• Academic research

Useless for:

• Predicting whether it will handle your refund policy correctly

• Knowing if it will escalate frustrated customers at the right time

• Determining if it respects your company’s tone of voice

App Evaluation (What You Actually Need)

Tests: “Does my customer support agent correctly process refunds, handle escalations, and follow our policies?”

This is what matters because your users don’t care if GPT-5 scored 95% on MMLU. They care if it solved their problem.

Your evaluators must be grounded in your business use case, not generic academic benchmarks. This means:

• Testing against your actual policies, not Wikipedia facts

• Using your real user queries, not synthetic textbook questions

• Measuring outcomes that impact revenue, retention, or safety

Benchmarks tell you which LLM is “generally smarter.” App evals tell you which version of your system works better for your users.

Don’t mistake one for the other.

Components of an Evaluator

Now that you know the types, let’s build one. Every evaluator has three components:

1. Reference Examples (Few-Shot Prompts)

These are the labeled examples from your dataset. They show the evaluator what “good” and “bad” look like for your specific task.

Remember from Article 2: the real power isn’t in the system prompt, it’s in these few-shot examples. They encode your domain expert’s judgment.

Example:

Example 1 - PASS

Input: “I need a refund for order #12345”
Output: “I’ve processed your refund. You’ll see the credit in 3-5 business days.”
Reason: Confirms action, sets timeline, professional tone.

Example 2 - FAIL

Input: “Can you waive the late fee on my account?”
Output: “I can help with that!”
Reason: Didn’t actually take action or explain next steps. Empty promise.

2. Metrics

The quantifiable measurement of quality. This can be:

• Objective: Did it call the right tool? Is the JSON valid? Is the response under 200 words?

• Subjective: Was it helpful? Was the tone appropriate? Did it follow the conversation flow?

For objective metrics, use code-based checks (fast, cheap, deterministic).

For subjective metrics, use LLM judges or human evaluation.

3. Rubrics

For subjective metrics, you need a rubric: explicit criteria that define what you’re measuring.

Bad rubric:
“Was the response helpful?”

(Too vague. Helpful how? To whom? Compared to what?)

Good rubric:
“Did the response: (1) correctly identify the user’s request, (2) provide a specific action or next step, (3) include a timeline or expectation, and (4) maintain professional tone?”

Rubrics force precision. They make subjective judgments repeatable. These criteria become part of your LLM judge’s system prompt.

Code-Based Evaluators: Fast, Cheap, Objective

Some checks are deterministic. Did the agent call refund_order()? Is the output valid JSON? Does it include a required disclaimer?

Use code for these. It’s faster, cheaper, and never gives you a different answer on the same input.

Use code-based evaluators for:

• Tool calls: Did it call refund_order() with the right parameters?

• Format checks: Is the output valid JSON? Is it under the character limit?

• Required elements: Does it include a disclaimer? Does it have a timestamp?

• Prohibited content: Does it contain banned phrases or leaked data?

Example (pseudocode):

def evaluate_refund_agent(trace):
  # Check if right tool was called
  if “refund_order” not in trace.tool_calls:
  return {”pass”: False, “reason”: “Didn’t call refund_order”}

  # Check if order_id parameter was provided
  params = trace.tool_calls["refund_order"].parameters
  if "order_id" not in params:
    return {"pass": False, "reason": "Missing order_id parameter"}

  # Check if response includes timeline
  if not any(word in trace.output.lower() for word in ["days", "week", "timeline"]):
    return {"pass": False, "reason": "No timeline provided to customer"}

  return {"pass": True, "reason": "All checks passed"}`

Code-based evaluators are:

• Fast: Milliseconds per check

• Cheap: No API costs

• Reproducible: Same input always gives same result

• Easy to debug: When they fail, you know exactly what broke

But they can’t handle nuance. They can’t judge tone, helpfulness, or conversational flow. For that, you need LLM judges.

These code-based evaluators work exactly like classic unit tests you’re already familiar with. They’re deterministic, fast, and easy to debug. That’s why you should always try to implement code-based checks first before reaching for LLM judges. If you can check it with code, do that. Only use LLM judges when code can’t capture what you need to measure.

LLM Judges: Flexible, Scalable, Nuanced

An LLM judge is an LLM that grades another LLM’s output. You give it the task, the output, and the evaluation criteria, and it returns a score with reasoning

LLM judges work in two modes: evaluating outputs against absolute criteria (is it helpful? professional? accurate?) or comparing outputs to reference answers when you have ground truth but need semantic understanding rather than exact string matching.

Use LLM judges for:

• Tone: Was it empathetic? Professional? Not condescending?

• Helpfulness: Did it actually answer the question or deflect?

• Conversation flow: Did it maintain context across turns?

• Reasoning quality: Did the agent’s plan make sense?

How it works:

1. You provide:

   • The input (user query)

   • The output (agent’s response)

   • The context (system prompt, retrieved docs, conversation history)

   • Evaluation criteria (what you’re checking for)

   • Few-shot examples (labeled passes and fails)

2. The LLM judge outputs:

   • A score (pass/fail or 0-1 scale)

   • A critique explaining why

Example prompt (simplified):

You are evaluating customer support responses. For each trace, output Pass or Fail 
with reasoning.

Evaluation criteria:
1. Did the response correctly identify the customer’s request?
2. Did it provide a specific action or next step?
3. Did it include a timeline or expectation?
4. Did it maintain a professional tone?

Here are examples of how a domain expert judged similar cases:

[Few-shot examples from your labeled dataset]

Now evaluate this trace:
Input: [customer query]
Output: [agent response]
Context: [system prompt, policies]

The judge generates:

FAIL

The response correctly identified the refund request (criterion 1: pass) and 
maintained professional tone (criterion 4: pass). However, it didn’t specify a next 
step beyond “we’ll look into this” (criterion 2: fail) and provided no timeline 
(criterion 3: fail). Customer is left waiting with no expectations set.

Strengths of LLM Judges

• Flexible: Handle open-ended tasks where code can’t

• Scalable: Grade thousands of traces automatically

• Explainable: Critiques show reasoning, helping debug failures

Weaknesses of LLM Judges

• Non-deterministic: Same input might get different scores across runs

• Expensive: Every evaluation is an API call

• Needs calibration: Must align with human judgment (we cover this in Article 5)

Making LLM Judges More Stable

1. Use the most capable model (e.g., Claude Opus, GPT-4o) + footnotes (4)

2. Add chain-of-thought reasoning before scoring (”Let’s think step-by-step...”)

3. Control for verbosity bias (normalize response lengths)

4. Run multiple trials and average scores for critical evals

5. Increase dataset size to at least 50-100 samples (reduces noise)

Common Mistakes (And How to Avoid Them)

Mistake 1: Not Providing Critiques

Wrong:
Score: 1

Right:
Score: 1

Critique: “Response correctly identified the refund request but didn’t provide a timeline. Customer left without expectations.”

Critiques are not optional. They’re how you debug failures and train better evaluators.

Mistake 2: Overly Terse Critiques

Wrong:
“Bad tone”

Right:
“Response used dismissive language (’just wait’) when customer expressed frustration about a delayed order. Should have acknowledged frustration and provided specific next steps.”

The critique should be detailed enough to serve as a few-shot example later.

Mistake 3: Missing Context

Don’t evaluate the output in isolation. Give the evaluator everything a human would see:

• The full conversation history (for multi-turn tasks)

• Retrieved documents (for RAG)

• System prompts (for understanding constraints)

• Tool call results (for agentic workflows)

If a human needs it to judge quality, the evaluator needs it too.

Mistake 4: Not Providing Diverse Examples

If all your few-shot examples are “customer angry, agent apologizes,” the judge won’t know how to handle “customer confused, needs technical explanation.”

Cover the failure modes you actually see in production.

Mistake 5: Using Ready-Made Metrics Without Validation

ROUGE, BLEU, BERTScore, etc. sound professional, but they might not correlate with your actual goal.

Before using any metric, validate it against human judgment on your specific task. If high ROUGE doesn’t mean “users are happy,” don’t optimize for ROUGE.

Mistake 6: Using 1-5 Scales Instead of Binary Pass/Fail

Wrong:
Score: 3.2 out of 5

Right:
Score: 0 (Fail)
Critique: “Response didn’t provide a timeline or next steps.”

Why it matters: A score of 3.2 is ambiguous. Is that good enough to ship? Should you fix it? Binary forces clarity. Either it passes your quality bar or it doesn’t. Scoring on a float scale (0.0-1.0) has the same problem - it leaves room for interpretation instead of forcing a clear decision.

When Should I Use Similarity Metrics (BERTScore, ROUGE, etc.)?

Short answer: Only for specific, narrow tasks where semantic overlap actually matters.

When They Work

Summarization: ROUGE measures how much of the source content appears in the summary. If your task is “don’t miss key facts,” ROUGE helps.

Translation: BLEU checks n-gram overlap with reference translations. Works when there’s a narrow acceptable output space.

Retrieval accuracy: BERTScore compares semantic similarity between retrieved chunks and expected documents.

When They Fail

Open-ended generation: Your AI agent says “I’ve refunded order #12345. You’ll see the credit in 3-5 days.” Reference says “Refund processed for order #12345, expect 3-5 business days.” Different words, same meaning. ROUGE fails.

Tone and helpfulness: Similarity metrics don’t measure if the tone was appropriate or if it actually helped the user.

Business outcomes: High similarity doesn’t mean the customer is satisfied, the sale closed, or the task completed.

The Rule

If your success criterion is “output should be semantically similar to the reference answer,” use similarity metrics.

If your success criteria are “user achieved their goal,” use app-level evaluators grounded in outcomes.

Advanced Metric Designs

Now let’s handle the hard cases: multi-turn conversations, complex workflows, and agentic systems.

Evaluating Multi-Turn Conversation Traces

A single-turn eval checks one input and one output. Multi-turn evals check entire conversations.

Challenges:

• Context must carry across turns

• Errors compound (one bad response derails the rest)

• You need to catch the first upstream failure, not downstream symptoms

Strategy:

1. End-to-end task success: Did the agent accomplish the user’s goal by the end?

2. Turn-by-turn checks: Evaluate each exchange individually

   • Did turn 3 maintain context from turn 1?

   • Did turn 5 escalate when the user got frustrated?

3. Failure attribution: When something breaks, find the first turn where it went wrong

Example (customer support conversation):

Turn 1:

User: “I need to return order #12345”

Agent: “Sure, I can help with that. What’s the reason for the return?”

Eval: Pass (acknowledged request, asked clarifying question)

Turn 2:

User: “It arrived damaged”

Agent: “I’ll process a refund. Expect 3-5 business days.”

Eval: FAIL (Skipped required step: didn’t offer replacement or ask for photos of damage)

Turn 3:

User: “Do I need to ship it back?”

Agent: “No, keep it.”

Eval: Pass (but only because Turn 2 already failed the workflow)

The first upstream failure is Turn 2. Everything after is a consequence.

Important: When evaluating any turn, provide all previous turns as context. Evaluating Turn 2? Include Turn 1. Evaluating Turn 3? Include Turns 1 and 2. The evaluator needs the full conversation history to judge whether context was properly maintained.

Evaluating Complex Multi-Step Workflows

Workflows have dependencies. Step 3 can’t succeed if Step 1 failed. Your evaluator needs to know this.

Example (flight booking agent):

Required sequence:

1. Search flights

2. Validate availability

3. Confirm payment

4. Book reservation

Bad eval: Check if all steps ran (yes/no)

Good eval: Check if steps ran in the right order, with correct dependencies

Evaluating Agentic Workflows

Agents don’t follow fixed scripts. They plan, reason, and adapt. This makes evaluation harder.

Two-phase approach (from Hamel Husain) (5):

Phase 1: End-to-End Task Success

Treat the agent as a black box. Did it meet the user’s goal?

Define precise success rules per task:

• Exact answer match (for factual tasks)

• Correct side-effect (database updated, email sent, file created)

• User satisfaction (thumbs up, complaint rate, retry rate)

Use human judges or well-aligned LLM judges. Focus on first upstream failures during error analysis.

Phase 2: Step-Level Diagnostics

Once you know which workflows fail, diagnose why.

Assuming you’ve instrumented your system to log tool calls and responses, score:

1. Tool choice: Was the selected tool appropriate?

2. Parameter extraction: Were inputs complete and well-formed?

3. Error handling: Did it recover from empty results or API failures?

4. Context retention: Did it preserve earlier constraints?

5. Plan quality: Does the agent’s plan match the task requirements?

Transition matrix analysis (Bryan Bischof’s approach):

Track which state transitions cause failures.

Example (text-to-SQL agent):

• GenSQL → ExecSQL: 12 failures

• DecideTool → PlanCal: 2 failures

This data-driven view shows where to focus debugging.

Session-level metrics:

• Task completion rate

• Step completion (did it finish the required steps?)

• Trajectory quality (did it avoid loops?)

• Self-aware failures (did it acknowledge limitations?)

Node-level metrics (per tool call):

• Tool correctness (right tool with right parameters?)

• Tool call accuracy (did the tool run without errors?)

• Output correctness (did the tool return valid results?)

System efficiency metrics:

• Latency (time to complete task)

• Token usage (cost per task)

• Tool calls per task (efficiency of plan)

These metrics layer on top of each other[6]. System efficiency ensures scalability. Session-level metrics validate goal achievement. Node-level metrics pinpoint root causes.

Bringing It All Together

Pick evaluators based on what you’re actually trying to measure, not what sounds impressive. Here’s how to decide which evaluator to use:

Can you check it with code?

Yes → Use code-based evaluators (tool calls, format checks, required elements)

No → Move to next question

Is there a single correct answer or narrow acceptable range?

Yes → Use reference-based evaluation (exact match, ROUGE, BLEU)

No → Move to next question

Are you comparing two versions?

Yes → Use pairwise comparison

No → Use direct scoring

Is the task subjective (tone, helpfulness, flow)?

Yes → Use LLM judges with rubrics and few-shot examples

No → Rethink your criteria (you might have missed a code-based check)

Is it a multi-turn or agentic workflow?

Yes → Use two-phase approach (end-to-end task success + step-level diagnostics)

No → Single-turn direct scoring

And remember: your evaluators are only as good as your dataset and few-shot examples. The system prompt matters less than you think. The examples matter more than you think.

Next Steps

You now know how to design evaluators that match your use case. You know when to use code, when to use LLMs, and when to combine both.

But here’s the critical question we haven’t answered: How do you know if your evaluator is actually working?

An evaluator who says everything is great when it’s not is worse than no evaluator at all. You need to validate that your automated judges align with human judgment before you trust them.

That’s what we’ll cover in Article 5: How to Evaluate the Effectiveness of the Evaluator.

Also, remember that this article is part of a 7-piece series on AI Evals & Observability. Here’s what’s ahead:

1. Integrating AI Evals Into Your AI App

2. Build an AI Evals Dataset from Scratch

3. Generate Synthetic Datasets for AI Evals

4. How to Design Evaluators ← You just finished this one

5. How to Evaluate the Evaluator

6. RAG Evaluation: The Only 6 Metrics You Need

7. Lessons from 6 Months of Evals on a Production AI Companion

See you next Tuesday.

Paolo Perrone

What’s your opinion? Do you agree, disagree, or is there something I missed?

Leave a comment

Most AI newsletters give you news. The AI Engineer gives you understanding.

One concept per week, explained from first principles: when to fine-tune vs. prompt vs. RAG, which vector database fits your workload, and how companies like DoorDash ship AI at scale.

Written for senior engineers and tech leads who build with AI, not just read about it.

The AI Engineer

Where software engineers become dangerously good AI engineers.

By Paolo Perrone

Go Deeper

Go from zero to production-grade AI agents with the Agentic AI Engineering self-paced course. Built in partnership with Towards AI.

Across 34 lessons (articles, videos, and a lot of code), you’ll design, build, evaluate, and deploy production-grade AI agents end to end. By the final lesson, you’ll have built a multi-agent system and a capstone project where you apply everything you've learned on your own.

Three portfolio projects and a certificate to showcase in interviews. Plus a Discord community where you have direct access to other industry experts and me.

Rated 4.9/5 ⭐️ by 290+ early students — “Every AI Engineer needs a course like this.”

Learn more

Not ready to commit? We also prepared a free 6-day email course to reveal the 6 critical mistakes that silently destroy agentic systems. Get the free email course.

Thanks again to Opik for sponsoring the series and keeping it free!

If you want to monitor, evaluate and optimize your AI workflows and agents:

Try Opik for free

References

1. Anthropic. (n.d.). Demystifying evals for AI agents. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

2. Evidentlyai. (n.d.). LLM-as-a-judge: a complete guide. https://www.evidentlyai.com/llm-guide/llm-as-a-judge

3. Evidentlyai. (n.d.). LLM evaluation metrics and methods. https://www.evidentlyai.com/llm-guide/llm-evaluation-metrics

4. OpenAI. (n.d.). Evaluation best practices. https://developers.openai.com/api/docs/guides/evaluation-best-practices

5. Husain, H. (n.d.). How do I evaluate agentic workflows? https://hamelhusain.substack.com/p/how-do-i-evaluate-agentic-workflows

6. Maxim. (n.d.). Evaluating agentic workflows: The essential metrics that matter. https://www.getmaxim.ai/articles/evaluating-agentic-workflows-the-essential-metrics-that-matter

7. Confident AI. (n.d.). LLM evaluation metrics: Everything you need for LLM evaluation. https://www.confident-ai.com/blog/llm-evaluation-metrics-everything-you-need-for-llm-evaluation

Images

If not otherwise stated, all images are created by the author.

This page is your Command Center. A clear map to the blueprints you need to move past "fancy demos" and ship production-grade AI.

Here’s how to find what you need ↓

📍Step 1: The Decoding AI Roadmaps

If you’re here for the specific architectures and mental models, start by exploring my past work. To keep things tidy, I’ve moved the full master index to a dedicated page where you can navigate the magazine at your own pace.

You can filter the entire archive by:

• Level: Beginner, Intermediate, Advanced.

• Collections: Foundations, Case Studies, Projects.

• Series: Per Topic End-to-End Blueprints.

Explore the Roadmaps →

📍Step 2: The Resource Library

If you’re looking to go deeper with more structured guides, here’s where to look next. While the weekly content is great for staying sharp, if you're ready to build a complete system from scratch without piecing together different articles, I’ve compiled the best of what I know into a few digital products.

Unlike the weekly posts, these include full codebases, video walkthroughs, and Q&A support to help you go from a blank IDE to a deployed system.

• The LLM Engineer’s Handbook: A framework for building LLM and RAG apps.

• Agentic AI Engineering Course: The end-to-end blueprint for designing, testing, and deploying autonomous agents.

• Full Course Catalog: Write real code. Ship AI that actually works.

Not sure what to pick? I also have a 6-day free email course on the critical design mistakes that silently break agentic systems. It boils down 2+ years of production experience into a simple mental model for building reliable agents that actually scale.

📩 Every Tuesday

You’ll get one actionable project, case study, or concept deep dive focused on the reality of shipping AI.

• Real-world: No bedtime stories, just hands-on content.

• Time-efficient: One free actionable tip in less than 8 minutes.

• Future-proof: Skills that will thrive in a future dominated by AI coding tools.

Check the latest insights →

💬 Keep in Touch

I’m building and sharing what works, and what doesn’t, every week. If you want to see the "work in progress" or the journey behind these systems, I also post here:

LinkedIn | X | GitHub | pauliusztin.ai

Happy learning,
Paul Iusztin

Images

If not otherwise stated, all images are created by the author.

That gap is exactly why I spent the last 9 months building an Agentic AI Engineering course with Towards AI. And here is what makes it different: we didn’t just teach how to build agents. We built two production AI systems, used them daily, and wrote the course with them.

Google and Gemini are already recommending it alongside courses from Coursera and DeepLearning.AI:

How This Course Was Built

Back in January 2025, Louis-François Bouchard (Co-Founder at Towards AI) reached out to me about creating a course on Agentic AI Engineering. I deeply respected Louis’s work in the AI space. So I said yes.

By April 2025, we had a team of five and one non-negotiable rule: we would only teach something we actually use ourselves. No toy examples. No throwaway demos.

We settled on an ambitious idea: a deep research agent and a writing workflow specialized in generating high-quality lessons and articles with text, code, images, diagrams, and references. We called them Nova and Brown.

The twist: we used Nova and Brown to write the course itself. Every lesson went through the same AI system we were teaching students to build. If something broke, we fixed it. Not for a demo, but because we needed it to work. That pressure forced us to build something production-ready, not just classroom-ready.

Nova and Brown are two MCP servers that can be orchestrated within a multi-agent system through Cursor, Claude Code, or any custom orchestrator. We created an AI system that writes about itself.

Learn more

What You Get

34 lessons that take you from foundations to deploying your own agent through articles, videos, and hands-on Notebooks. You will learn tool calling, ReAct loops, context engineering, structured generation, memory systems, RAG, planning and reasoning architectures, human-in-the-loop feedback, and CI/CD deployment:

• Self-paced with monthly live kick-off sessions so you can go at your own speed without losing momentum.

• 4 parts: Foundations (multiple smaller projects), two end-to-end complex projects, LLMOps (evaluation, observability, auth, deployment), and a final capstone project you implement yourself.

• Real code, not notebook-only demos. The teaching happens through Notebooks, but the code is structured as two Python modules (Nova and Brown). You import from the modules into Notebooks for a structured learning experience.

• Fundamentals over frameworks. We wrote as much as possible from scratch because tools change constantly. The course focuses on design principles and patterns you can replicate in any tool. Key tools used: LangGraph, LangChain, Gemini, FastMCP, Cursor/Claude Code, Opik, Perplexity, and GCP.

• Discord community with Q&A support and a completion certificate.

Who Is This For?

Engineers who want to go deep on AI agents, not skim the surface. If you are a software engineer, ML engineer, or data scientist who has played with LLMs but never built a multi-step agent that actually works in production, this is for you.

You should be comfortable with Python, have basic familiarity with LLMs, Docker, and cloud. And above all: a builder mindset.

Learn more

Early-bird pricing: $449 for lifetime access — limited to the first 100 seats!

💡 Not sure yet? We open-sourced the code on GitHub and made the first 6 lessons free.

What Students Are Saying

We sold 150 pre-release slots to build the course with a real audience. The result: 25 five-star reviews. Not from our own biased impression, but from students who went through the material.

As one reviewer put it: “goes far beyond theory, providing deep, practical experience” with real-world constraints rather than flashy demos.

Sean Myers, Principal Analyst at Columbia, already earned the first completion certificate:

Here you can learn more:

Learn more

Paid Subscribers

For paid subscribers, we are offering 20% off. For the discount code, DM me on Substack or comment on this post.

We will soon create a paid subscribers’ perks page with more offers. But for now, let’s keep it simple.

Looking forward to your feedback on the course and seeing you next Tuesday!

Paul

🧐 Everyone says you need AI evals. Few explain how to actually build them and answer questions such as…

How do we avoid creating evals that waste our time and resources? How do we build datasets and design evaluators that matter? How do we adapt them for RAG? ...and most importantly, how do we stop “vibe checking” and leverage evals to actually track and optimize our app?

This 7-article series breaks it all down from first principles:

1. Integrating AI Evals Into Your AI App

2. Build an AI Evals Dataset from Scratch

3. Generate Synthetic Datasets for AI Evals ← You are here

4. How to Design Evaluators

5. How to Evaluate the Evaluator

6. RAG Evaluation: The Only 6 Metrics You Need

7. Lessons from 6 Months of Evals on a Production AI Companion

By the end, you’ll know how to integrate AI evals that actually track and improve the performance of your AI product. No vibe checking required!

Let’s get started.

Generate Synthetic Datasets for AI Evals

In the previous article, you learned how to iteratively build an evals dataset using the error analysis framework. You started from production traces, labeled them, fixed errors, and grew your dataset over time. But what if you lack production traces?

What if your production data only covers a fraction of the features, personas, and edge cases your app supports? Synthetic data fills the gaps that production data alone cannot cover.

When I was building Nova, the deep research agent for my Agentic AI Engineering course, I hit this exact wall. I built an evaluation layer with binary metrics across dozens of dimensions. The metrics were solid, and the LLM judge was calibrated.

But then I needed test data. The agent lacked real users, real generated articles, and traces. I started by manually writing test inputs. After a painful week, I had maybe 15 examples.

They all reflected my own biases. I was testing the same happy path over and over. Entire categories of failure modes went completely untested.

The most time-consuming bottleneck wasn’t building the judge. It was generating enough diverse, realistic test inputs. That experience taught me that structured synthetic data generation unlocks your entire evals pipeline.

Most teams fire off a single generic prompt to create test inputs. The result is a homogenous, shallow dataset where most examples look identical. The LLM converges on the most generic patterns, causing mode collapse.

You end up testing the same happy path over and over. You need test data, but you lack sufficient or diverse production traffic. Naively generating synthetic data produces datasets that are repetitive and miss your business use cases.

A structured approach gives you control over the distribution of your test inputs. You achieve this by thinking in terms of dimensions, anchoring it in your business use case, and applying targeted strategies.

In this article, you will learn:

• When to rely on synthetic data generation.

• Why you should only generate user inputs.

• How to use dimensions to avoid mode collapse.

• Strategies to expand existing production data.

• Approaches for agents, RAG, and deterministic tasks.

Before digging into the article, a quick word from our sponsor, Opik. ↓

Opik: Automated Agent Optimization Using Your Evals Data (Sponsored)

This AI Evals & Observability series is brought to you by Opik, the LLMOps open-source platform used by Uber, Netflix, Etsy, and more.

We use Opik daily across our courses and AI products. Not just for observability, but now to automatically optimize our agents’ prompts using the same datasets and metrics we already have in the platform.

You are learning how to build diverse synthetic datasets to evaluate your AI app. But once you have those datasets and metrics, why stop at measuring quality? Opik’s agent optimizer closes the loop. It uses your eval dataset to automatically improve your prompts. Here is why we love it:

• Same datasets, zero extra setup — Opik’s optimizer reuses the exact datasets, metrics, and tracing you already have. Quick start guide.

• Six optimization algorithms — Choose from strategies like HRPO (our favorite), which performs root-cause analysis on failures and proposes targeted fixes, or evolutionary optimization to explore diverse prompt structures. See all algorithms.

• No-code Optimization Studio — For quick iterations, run optimization directly from the Optimization Studio UI. Start from your prompt, pick your dataset, choose an algorithm, and watch Opik test prompt variations against your metrics in real time.

Opik is fully open source and integrates with OpenAI, Anthropic, Gemini, and 100+ providers. Start optimizing your agents:

Optimize your AI agents

↓ Now, let’s move back to the article.

When Do We Need Synthetic Data?

Before you have any production data, you face the cold start problem. You might be building a new feature or preparing for a launch. You need to simulate months of traffic in hours to ensure a reliable initial release. You cannot wait for real users to find your bugs. Synthetic data lets you test your application before day one.

Sometimes your app is live, but you lack enough production data. You might have 50 traces instead of 5,000. The error analysis framework needs enough examples to surface recurring patterns. Synthetic data supplements your real traces.

Other times, your data lacks diversity. You might have plenty of production traces, but they cluster around a few common use cases. Most users ask the same types of questions. You end up with almost no examples of edge cases, adversarial inputs, or minority user personas. Synthetic data lets you deliberately target the underrepresented regions of your input space.

Now that you know when synthetic data is necessary, let’s understand the core principle behind how it works. Most people get this fundamental concept wrong.

Understanding the Core Principles

The single most important principle is that you generate only the user inputs. These queries, messages, or requests should be as diverse as possible. They must cover all your business use cases, edge cases, and user profiles.

You do not generate the intermediate steps or the final outputs. The whole point of your evals dataset is to capture how your actual system behaves. Synthetic outputs would test a fiction, not your real app.

Instead, you let your real app produce the traces. First, you generate diverse synthetic inputs. Then, you feed all generated inputs into your AI app as if they were real user requests.

You track every trace using your observability platform, such as Opik. This captures all intermediate tool calls, model reasoning, and final outputs. You then pull these full traces to create your synthetic evals dataset.

Finally, you apply the error analysis framework we learned in Article 2. You label the data with pass/fail judgments, fix errors, build evaluators, and iterate. Your synthetic dataset contains real system behavior triggered by synthetic inputs.

This makes it a valid proxy for production. Since the app handles processing and tool calls, the entire challenge reduces to a single thing:

You must create diverse, realistic, business-grounded inputs.

Thus, the entire problem reduces to generating input data. Let’s explore concrete strategies for doing so, starting with the most fundamental approach.

Seeing Your Inputs as Dimensions

When generating input data, using a generic prompt is a mistake. You have zero control over the distribution of edge cases the LLM generates. The LLM will converge on the most generic, repetitive patterns.

Instead, think about a few key dimensions that matter for your application. Model them as tuples to serve as the seeds for your generation process. A common dimension tuple includes the persona, feature, scenario, and input modality.

The persona defines the different user types who interact with your app. You get inputs from an impatient customer, a technical expert, or a confused first-time user.

The feature represents the different capabilities of your app that you want to test. Examples include answering emails, generating a meeting summary, or drafting an article.

The scenario defines the specific failure modes or edge cases you want to stress-test. This includes contradictory instructions, garbled input, or outdated information.

The input modality is the format through which the input arrives. This could be plain text, a forwarded email thread, a voice transcript with filler words, or a pasted spreadsheet snippet.

If you define 3 personas, 5 features, 10 scenarios, and 3 input modalities (text, image, documents), and combine them all, you get a maximum of 450 unique data seeds. Each seed is a specific combination that drives a targeted, diverse input.

For each seed, you craft a generation prompt. This prompt includes the dimension values, context about your app, and the scenario’s built-in failure assumption.

Embedding the failure assumption directly in the scenario is highly effective. You tell the LLM exactly what failure to target and what correct behavior looks like [1]. This makes your synthetic inputs far more precise.

Here are six dimension seeds with their generation prompts for an email and messaging assistant:

Your dimensions will vary depending on your business use case. You might have entirely different dimensions, such as language or urgency level.

How much data do you need?

At a minimum, generate enough data so that you have at least one example for each combination of dimensions. Keep generating more data until you stop seeing new failure modes. A simple chatbot might need 200 examples, while a complex agent might need over a thousand.

Does this actually work?

You might wonder if synthetic data actually works. From my experience, if well-guided, LLMs are highly capable of generating excellent, diverse examples of user prompts. Synthetic data is the fastest way to build a meaningful evals dataset early on.

Dimension-based generation works great when you start from scratch. But what if you already have some production data and want to expand it?

When Having Some Production Data

When you have production data, identify your failed or most difficult interactions to use as seeds. If a user input caused your system to fail, generate many variations of that specific input.

You generate multiple variations of the same input with the same underlying semantics to stress-test your app’s consistency. You can vary the phrasing, the level of aggression, or the ambiguity. This ensures your system never fails the same way twice. This method is known as Metamorphic testing.

Suppose your agent fails when users send multi-part questions in a single message. You take this real failed trace, combine it with our multi-dimensional strategy, and ask an LLM to generate 20 variations. The resulting inputs target the same failure class with enough variation to test your fix reliably.

Another method is evolutionary complexity, also known as Evol-Instruct. Originally introduced by the WizardLM researchers to generate synthetic training data for LLMs [7], it transfers remarkably well to generating evaluation data.

The core problem is the same: producing diverse, progressively complex instructions from a small set of seeds. It uses an evolutionary paradigm to transform simple seed inputs into more complex, realistic ones [1].

Evol-Instruct is based on 3 core steps:

1. In-depth evolving takes a simple instruction and increases its complexity. It adds constraints, deepens the subject matter, or increases reasoning requirements. A simple order status query evolves into a complex rerouting request.

2. In-breadth evolving generates completely new, diverse instructions. This ensures the evaluation suite covers a broad range of topics. While in-depth evolving makes existing inputs harder, in-breadth evolving widens the dataset.

3. Elimination evolving is a filtration step. A critic LLM evaluates evolved instructions and discards those that provide no information gain or are nonsensical. This keeps the quality high as complexity grows.

These methods work well for single-turn interactions. But what about AI agents that handle multi-step, multi-turn conversations?

When Building Agents

For complex AI agents that plan and execute multi-step workflows, evaluation moves beyond single-turn queries. You need to generate an entire conversation.

To evaluate these systems, you set up a dual-agent dynamic: a tester agent simulating the user, and your actual app agent. The tester agent dynamically generates synthetic inputs turn by turn, reacting to your app’s responses just like a real user would [3]. For example, a tester agent playing a frustrated customer might escalate their tone if the first response is vague or pivot to a different request mid-conversation.

Implementing this tester agent uses the exact same ideas as single-turn generation. You define dimensions like personas and scenarios to impersonate, but you simply run the generation iteratively for each conversation turn.

Some recent research from Nov 2025 shows that agents achieve over 90% accuracy on single-step tasks, but conversation correctness drops to 10-15% on full conversations [3]. This makes multi-turn flow evaluation a necessity for production reliability.

Multi-turn interactions are one specialized case for synthetic data. Another common scenario is information retrieval, where your app searches a knowledge base before responding.

When Doing RAG

When your AI app retrieves information from a knowledge base before generating a response is known as Retrieval-Augmented Generation (RAG). In this scenario, you can use a reverse workflow to create ground-truth datasets. Instead of the standard retrieval flow, you start with the knowledge base and work backwards [4].

You start by taking your documents, PDFs, or structured data. You use an LLM to extract key facts, procedures, numbers, or policies from a specific document chunk that’s part of your knowledge base.

The LLM then generates a realistic user question that can only be answered using that specific chunk. Because the question is derived directly from the source material, you know exactly which document chunk should be retrieved. Which means you can easily generate the right answer as well.

This guarantees perfect alignment between the input, the expected retrieval context, and the expected output. You get a complete ground-truth triplet containing the question, relevant context, and expected answer.

To create more diversity within the dataset, you can use the same strategy to define multiple dimensions and mix up question styles and complexity levels to avoid a shallow dataset.

This reverse synthesis works because you can derive questions and answers from documents. But what about purely deterministic tasks where the correct answer can be computed exactly?

For Deterministic Testing

For tasks with deterministic correct answers, you can use your system’s schema or rules to generate both the input and the ground truth. This applies to structured data extraction or calculations, such as text-to-SQL, JSON, code, or math.

You work from the answer backward to the question, similar to the reverse workflow for RAG. However, instead of text, you use schemas, databases, or rule sets [6].

Suppose you want to generate (text, SQL) tuples. You have a database with tables for customers, orders, and products. You use your database schema to programmatically generate valid SQL queries of varying complexity. These SQL queries serve as your ground truth.

You then use an LLM to translate each SQL query back into a natural language question. A query selecting customers with pending orders over a specific amount becomes a plain English question.

Your evals dataset now has a natural language input and the correct SQL mapping. You can test whether your system generates the right query and returns the right data.

Next Steps

Building an evals dataset from production traces alone has limits. Synthetic data solves the cold start problem and fills coverage gaps.

Synthetic data generation is not about blindly asking an LLM to create test cases. It is about structuring your inputs as dimensions, anchoring them in your business use case, and applying the right strategy. The result is a diverse, controlled dataset that you can feed into your error analysis framework.

Now you know how to build your evals dataset from both production and synthetic data. In the next article, we will show you the right way to build your evalautor(s).

Also, remember that this article is part of a 7-piece series on AI Evals & Observability. Here is what’s ahead:

1. Integrating AI Evals Into Your AI App

2. Build an AI Evals Dataset from Scratch

3. Generate Synthetic Datasets for AI Evals ← You just finished this one

4. How to Design Evaluators

5. How to Evaluate the Evaluator

6. RAG Evaluation: The Only 6 Metrics You Need

7. Lessons from 6 Months of Evals on a Production AI Companion

See you next Tuesday.

Paul Iusztin

What’s your opinion? Do you agree, disagree, or is there something I missed?

Leave a comment

Enjoyed the article? The most sincere compliment is to share our work.

Share

Go Deeper

Go from zero to production-grade AI agents with the Agentic AI Engineering self-paced course. Built in partnership with Towards AI.

Across 34 lessons (articles, videos, and a lot of code), you’ll design, build, evaluate, and deploy production-grade AI agents end to end. By the final lesson, you’ll have built a multi-agent system and a capstone project where you apply everything you’ve learned on your own.

Three portfolio projects and a certificate to showcase in interviews. Plus a Discord community where you have direct access to other industry experts and me.

Rated 4.9/5 ⭐️ by 290+ early students — “Every AI Engineer needs a course like this.”

Learn more

Not ready to commit? We also prepared a free 6-day email course to reveal the 6 critical mistakes that silently destroy agentic systems. Get the free email course.

Thanks again to Opik for sponsoring the series and keeping it free!

If you want to monitor, evaluate and optimize your AI workflows and agents:

Try Opik for free

References

1. Confident AI. (n.d.). The Definitive Guide to Synthetic Data Generation Using LLMs. Confident AI. https://www.confident-ai.com/blog/the-definitive-guide-to-synthetic-data-generation-using-llms

2. Husain, H. (n.d.). Using LLM-as-a-Judge For Evaluation: A Complete Guide. Hamel’s Blog. https://hamel.dev/blog/posts/llm-judge/#example-llm-prompts-for-generating-user-inputs

3. Zendesk. (n.d.). Building realistic multi-turn tests for AI agents. Zendesk. https://www.zendesk.com/au/blog/zip1-building-realistic-multi-turn-tests-for-ai-agents/

4. Evidently AI. (n.d.). How to create LLM test datasets with synthetic data. Evidently AI. https://www.evidentlyai.com/llm-guide/llm-test-dataset-synthetic-data

5. Langfuse. (n.d.). Synthetic Dataset Generation for LLM Evaluation. Langfuse. https://langfuse.com/guides/cookbook/example_synthetic_datasets

6. DeepEval. (n.d.). Generate Synthetic Test Data for LLM Applications. DeepEval. https://deepeval.com/guides/guides-using-synthesizer

7. Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., & Jiang, D. (2023). WizardLM: Empowering Large Language Models to Follow Complex Instructions. arXiv. https://arxiv.org/abs/2304.12244

Images

If not otherwise stated, all images are created by the author.

Today, I’m taking a short break from the technical deep dives to write something a little different.

Read more

🧐 Everyone says you need AI evals. Few explain how to actually build them and answer questions such as…

How do we avoid creating evals that waste our time and resources? How do we build datasets and design evaluators that matter? How do we adapt them for RAG? ...and most importantly, how do we stop “vibe checking” and leverage evals to actually track and optimize our app?

This 7-article series breaks it all down from first principles:

1. Integrating AI Evals Into Your AI App

2. Build an AI Evals Dataset from Scratch ← You are here

3. Generate Synthetic Datasets for AI Evals

4. How to Design Evaluators

5. How to Evaluate the Evaluator

6. RAG Evaluation: The Only 6 Metrics You Need

7. Lessons from 6 Months of Evals on a Production AI Companion

By the end, you’ll know how to integrate AI evals that actually track and improve the performance of your AI product. No vibe checking required!

Let’s get started.

Build an AI Evals Dataset from Scratch

In the previous article, you learned where, when, why and what AI Evals are. You saw the three core scenarios (optimization, regression, production monitoring) and the tech stack. But knowing where to use evals is only half the battle. You still need the actual dataset and evaluators to run them. That is what the rest of the articles from the series are about. The “how”.

For example, after shipping Brown (the writer agent from our Agentic AI Engineer course capstone project), I've been actively using it to write articles and lessons (even this one, haha), thus generating a lot of “production” traces. However, I had no structured way to evaluate them. I would look at a few outputs, tweak a prompt, and hope for the best. It wasn’t until I sat down, pulled 50 traces, and started writing notes on what went wrong with each one that I realized most of my failures fell into just 3–4 categories. That simple exercise of looking at the data changed everything. It told me exactly what to measure and what to fix first.

In reality, is not that simple, but not far from it. That’s what I want to teach you in this article.

Instead, most teams skip straight to building fancy evaluation dashboards or crafting elaborate scoring criteria without ever looking at their data. As Hamel Husain puts it: “Many teams make the mistake of crafting elaborate eval criteria without first looking at the data” [2]. This leads to two common traps: creating irrelevant criteria that waste resources on low-probability defects, or setting unrealistic criteria that the technology isn’t ready for.

You know you need AI evals, but you likely don’t have a dataset, you don’t have an evaluator, and you don’t know where to start. Building everything from scratch feels overwhelming. That is why many people start using generic tools and metrics, which is another huge mistake.

The solution is the error analysis framework. This is a step-by-step flywheel: start small, let the data guide you, and iteratively grow your dataset and evaluators. You do not need hundreds of examples or a perfect system on day one. You need 20–50 real traces and the discipline to look at them carefully, which you can easily start from day zero of your project.

In this article, we will cover:

1. How to create and format your initial dataset from production or synthetic traces.

2. How to manually label your data.

3. How to fix errors and grow your dataset with regression tests.

4. The iterative process of building and aligning an LLM judge.

5. How to perform systematic error analysis to cluster and prioritize fixes.

6. When to move from generic evaluators to specialized ones.

Before digging into the article, a quick word from our sponsor, Opik. ↓

Opik: Open-Source Observability for Your Multimodal AI Agents (Sponsored)

This AI Evals & Observability series is brought to you by Opik, the LLMOps open-source platform used by Uber, Netflix, Etsy, and more.

We’re proud to partner with a tool we actually use daily across our open-source courses and real-world AI products. Why? Because it makes evaluating multimodal AI apps as easy as evaluating text ones.

AI apps are no longer just text-in, text-out. They process images, generate videos, parse PDFs, and more. Monitoring and evaluating all of that used to be painful. With Opik, it’s not. Here is why we love it:

• Trace everything — Opik renders images, videos and PDFs directly inside your traces. No more guessing what your model actually saw or generated. We use this daily, and it changed how we debug multimodal pipelines.

• Zero-friction multimodal evals — Add image URLs or upload files directly in the UI, then run LLM-as-a-Judge evaluations on them. Opik auto-detects vision-capable models (GPT-4o, Claude 3+, Gemini) and warns you if the model doesn’t support vision.

• Video generation? Traced automatically — Wrap your OpenAI client in one line, and Opik tracks the full Sora workflow: creation, polling, download, and logs the generated video as an attachment. Full visibility, minimal setup. Guide here.

Opik is fully open-source and works with custom code or most AI frameworks. You can also use the managed version for free (with 25K spans/month on their generous free tier). Learn more about evaluating multimodal traces:

Evaluate multimodal traces

↓ Now, let’s move back to the article.

Create the AI Evals Dataset

Before you can evaluate anything, you need a evals dataset. This is a collection of examples that represent how your app should behave. This is the foundation on which everything else builds. Start small, as 20–50 examples are enough, and grow it over time. As Anthropic recommends: “20-50 simple tasks drawn from real failures are a great start” [3].

The highest-value source for your evals dataset is real production traces. These are actual user interactions with your app. These reflect genuine usage patterns, edge cases, and failure modes that you could never fully anticipate upfront.

If you are pre-launch or have limited production data, start with the manual checks you already run during development. These are the behaviors you verify before each release, and common tasks end users try.

You must log everything: user input, system prompt, model output, tool calls, retrieved documents, and metadata such as channel, timestamp, and user ID. Make it easy to browse, filter, and search these traces. You can easily log these traces using observability/LLMOps tools such as Opik (which we always use) [4].

Aim for 50–100 traces initially. In early development, each change has a noticeable impact, so small sample sizes work fine. More mature systems need larger datasets.

In case you already have tons of production traces, you need to properly sample them.

More advanced sampling strategies include outlier detection (sorting by response length, latency, or tool calls and reviewing extremes), user feedback signals (prioritizing traces with negative feedback or escalations), metric-based sorting (using generic metrics as exploration signals), stratified sampling (grouping by user type, feature, or query category and sampling from each), and embedding clustering (generating embeddings of queries, clustering them to reveal natural groupings, then oversampling small clusters for edge cases).

If you don’t have enough production data, you can automatically create test examples using an LLM to generate synthetic user inputs across different dimensions like features, scenarios, and user personas. We will explore synthetic data generation in detail in Article 3.

Every example in your dataset should have a consistent structure:

1. Input: The user query or request.

2. Output: The final output of the agent.

3. Context: Any additional information the system had access to, such as retrieved documents, conversation history, or system prompts.

4. Trace Spans: The intermediate steps, such as tool calls (API calls, search operations), model calls or any other relevant step (e.g., mapping to Pydantic).

This structure ensures your evaluators have everything they need to judge whether the system performed correctly. It also makes your dataset portable across different evaluator types, from simple ones that just check whether a specific tool was called to more complex LLM Judges.

🔎 Suppose you are building a vertical assistant agent that drafts replies to work emails and Slack messages. You have been using it in production for two weeks and have logged traces. Here is how one formatted dataset entry looks like:

• Input: A Slack message in #product from your Product Manager: “@you Can you estimate how long it would take to add CSV export to the reports page?”

• Output: The agent’s generated reply: “Based on the current codebase, I’d estimate 3–5 days, including testing. The data layer is already structured for it — the main work would be the export logic and UI. Want me to write up a quick ticket with more detail?”

• Context: The system prompt instructs the agent to match workplace communication norms, the user’s role (engineer), the channel (#product), and relevant prior messages in the thread.

• Trace Spans: (1) A model call to classify the message type and urgency, (2) a retrieval step pulling relevant codebase context about the reports page, (3) a model call generating the draft reply.

You pull 50 such traces. These include a mix of emails and Slack messages across scenarios such as executive requests, colleague questions, and phishing attempts. You format each one this way.

Now that you have a dataset, the next step is to label it. You need to decide, for each example, whether the output is good enough.

Manually Label Your Dataset

To judge whether each example is “good” or “bad,” you need to appoint one or multiple domain experts. To avoid inconsistencies. The less, the better. The domain expert should be someone who genuinely understands your use case to lead the labeling process.

This step is key! Why? Because this person’s judgment becomes the definitive source of truth. In other words, how it labels your dataset will have a cascading effect on everything else.

For each example in your dataset, the domain expert makes a binary judgment: Pass or Fail. Do not use a 1–5 scale or letter grades. Just pass or fail.

Binary decisions force clarity. A score of “3.2 out of 5” is hard to interpret and even harder to act on [6]. Pass/fail forces you to articulate exactly what “good enough” means and creates actionable insights. If something fails, you know it needs fixing. If it passes, you move on.

For every judgment, especially failures, the domain expert must write a short critique explaining why it failed or passed. These critiques are gold. They capture the expert’s reasoning, surface patterns in what goes wrong, and later become a central piece in the few-shot examples you feed to your LLM judge.

Even if your labels are binary, using critiques adds detail, and instead of vague numbers like 3.4/5, a clear explanation highlights exactly what went well or what went wrong. Also, they indirectly act as instructions to your LLM judge. Reason why they are so important when adding them as few-shot examples.

Do not try to catch every single mistake in a trace. Find the first thing that went wrong, the most upstream error, and move to the next example. The goal is to surface recurring failure patterns, not to write a detailed report for each trace. However, do not be too concise. Each critique should be detailed enough to later serve as a few-shot example for your LLM judge.

With Opik’s API or MCP Server, you can easily extract traces into a spreadsheet, a simple internal tool, or your custom annotation tool, as discussed in Article 1, to do this. Display the input, the system’s output, and all the context side by side. Make labeling as frictionless as possible.

🔎 Consider our email/Slack assistant example. Your domain expert sits down with the first 50 traces and labels each one:

• Trace #5 — FAIL: A vendor sent four specific technical questions about API integration. The agent replied: “I’ll look into these and get back to you.” Critique: “Failed to answer any of the four specific questions. Gave no timeline for follow-up. A vendor expecting technical answers got a vague brush-off. This damages the partnership.”

• Trace #7 — PASS: The CFO approved a $50K budget reallocation and asked the user to loop in HR. The agent replied by restating the action items, confirming it would initiate the transfer and loop in HR, and promising a confirmation. Critique: “Accurately restates all action items from the CFO’s approval, commits to the next steps, and matches the professional tone expected for executive communication.”

• Trace #6 — FAIL: An obvious advance-fee scam email from “Prince Makumba” offering $8.5M. The agent replied: “Could you provide more details about this inheritance?” Critique: “Engaged with a textbook scam email instead of ignoring it. Any reply validates the scammer’s target. Expected behavior: no reply.”

You now have a labeled dataset with pass/fail judgments and critiques. The natural next step: fix the obvious problems you’ve just discovered.

Manually Fix Errors

The labeling process will reveal generic, often simple issues. Examples include a missing instruction in a prompt, a broken tool call, or a formatting problem. Fix these before doing anything else. Do not build an evaluator for something you can just fix right now.

As Hamel Husain recommends, address obvious errors discovered during review before building judges. The point of the flywheel is product quality, not a pretty eval suite [4].

After fixing, re-run your system with the same inputs and potentially new ones to generate fresh outputs. Label the new outputs. Did your fixes work? Did they introduce new problems? This is the inner loop of the flywheel: create, label, fix, and repeat. Each iteration improves your system and enriches your dataset.

Every iteration adds more labeled examples to your dataset. Failed examples that you have now fixed become regression test cases. They ensure old bugs do not come back. New examples expand coverage. Over time, your dataset becomes a living artifact that captures the accumulated knowledge of what “good” and “bad” look like for your specific app. This is the foundation for everything that follows. Aim for continuous growth: start with 20–50, grow to 100+, and keep adding as you discover new failure modes in production.

🔎 Back to our simple agent example that answers professional emails or Slack messages. During labeling, the domain expert flagged several traces where the agent replied to obvious phishing and scam emails.

The fix was straightforward: add an explicit instruction to the system prompt telling the agent to never reply to messages from unrecognized external senders requesting money, credentials, or personal information. It should flag them as suspicious instead. After applying the fix, the expert re-runs the same scam inputs through the updated system. The agent now correctly produces no reply for all of them. These previously-failing traces become regression test cases. This ensures this class of errors never returns, even after future prompt changes.

At some point, manually labeling every example doesn’t scale. That’s when you need an automated evaluator to do the heavy lifting for you.

Iteratively Build Your Evaluator

You have been labeling by hand. That worked for the first 50–100 examples, but now you want to evaluate thousands of traces automatically. You need an evaluator. This is a system that can judge outputs without a human in the loop.

The key insight is to build it iteratively using the human-labeled data you have already collected, rather than designing evaluation criteria from scratch.

Before building your evaluator, split your labeled dataset into subsets:

• Train: The examples you will use to build and tune your evaluator (e.g., as few-shot examples in your LLM judge prompt).

• Dev: The examples you will use to check whether your evaluator is working while you iterate on it.

• Test: The examples you set aside and never touch until you are ready for a final evaluation of your evaluator’s quality.

We will cover the details of how to split effectively and how to evaluate the evaluator’s effectiveness in Article 5. For now, the key idea is: do not train and test on the same data. Exactly how we do when training any other AI model.

Due to the nature of AI app outputs, which are non-deterministic, unstructured, and subjective, the most popular approach is to use an LLM to grade another LLM’s output. This is known as an “LLM judge” or “LLM-as-a-judge”.

At this stage, build one binary LLM judge that runs across your entire dataset. Think of it as a binary classifier: for each trace, it returns Pass (1) or Fail (0) plus a critique explaining its reasoning. Not a 1–5 scale. Not a letter grade. Just pass/fail with a written justification. This mirrors exactly what your domain expert did during manual labeling.

Binary judgments are clear, actionable, and easy to aggregate. A score of “3.2 out of 5” is hard to interpret and even harder to act on. Pass/fail forces clarity and creates actionable insights.

The real power lies in your few-shot examples and dataset, not your prompt. This is a counterintuitive but critical insight: the system prompt for your LLM judge can be almost neutral. Just specify what the task is, the expected output format (pass/fail + critique), and a few core steps. Keep it simple. The real guidance comes from the few-shot examples you include in the prompt. These are the labeled examples from your dataset with their critiques. These examples encode your domain expert’s judgment, show the LLM what “good” and “bad” look like for your specific use case, and steer the judge far more effectively than elaborate prompt instructions ever could. Your dataset is the secret weapon, not your system prompt. More on this in Article 4.

To build it, start with the critiques your domain expert wrote during manual labeling. Select representative pass and fail examples. These become the few-shot examples in your judge prompt. Test the judge against your dev set and iterate until it mostly agrees with your domain expert’s labels.

Not everything needs an LLM judge. For anything that can be checked with simple logic (true/false or numeric), use code-based checks. Examples include checking whether the response included a required disclaimer, whether it is within the word limit, whether it returned valid structured output, or whether it called the right tool.

Code-based evaluators are faster, cheaper, and more reliable than LLM judges for objective checks. Reserve LLM judges for subjective or nuanced checks. These include tone, helpfulness, conversational flow, or quality of handoffs, where correctness is hard to express in code. More on this in Article 4. For now, the rule of thumb is: use code when you can, use an LLM judge when you must.

Your evaluator is only useful if it agrees with your domain expert. Run the evaluator on the dev set and determine how often it agrees with the human. Create an agreement matrix comparing the human label to the evaluator’s label. If the evaluator says “Pass” when the human said “Fail” (or vice versa), refine the evaluator’s prompt or logic until the agreement is high enough to trust. Be aware that raw agreement can be misleading with imbalanced datasets. More on this in Article 5. For now, the key idea is: always validate your automated evaluator against human judgment before trusting it.

“Many teams make the mistake of crafting elaborate eval criteria without first looking at the data” [2]. This quote from Hamel Husain captures the core philosophy of the error analysis framework. Designing evaluation criteria in a vacuum. Without first reviewing your actual traces and failure modes, you risk creating irrelevant criteria that waste resources on low-probability defects or unrealistic criteria. The solution is to put the data and metrics first, not preset criteria or LLMs. Let the failure modes you discover through manual review and error analysis drive what you evaluate.

🔎 Back to our example. Let’s assume you select 15 labeled traces with their critiques from your dataset and use them as few-shot examples in your LLM judge prompt. For instance, you include the vendor email trace (Fail — “didn’t answer any of the four specific questions, gave no timeline”) and the CFO budget trace (Pass — “accurately restates action items, commits to next steps, matches professional tone”).

The judge's prompt simply says: “You are evaluating whether an AI email/Slack assistant produced an appropriate reply. For each trace, output Pass or Fail with a critique explaining your reasoning. Here are examples of how a domain expert judged similar traces:” followed by those few-shot examples. You run this judge on your dev set of 20 traces. It agrees with the domain expert’s labels on 18 out of 20. The two disagreements reveal that the judge is too lenient on vague responses to multi-part questions. You add another few-shot example covering that pattern, and agreement improves.

You now have automated evaluators aligned with human judgment. The next step is to run them on new data and analyze the errors they find.

Doing Error Analysis

Your evaluator is running. It is flagging failures across hundreds or thousands of traces. But a list of pass/fail results is not enough. You need to understand which errors are occurring, how often, and which to fix first. This is error analysis: the most important activity in evals. It is the systematic process of clustering, ranking, and acting on the failures your evaluators surface. It helps you decide what evaluators to create in the first place, allowing you to identify failure modes unique to your application and data.

Sample a fresh batch of production traces (or new synthetic data) from your observability layer, such as Opik, that your evaluator hasn’t seen before. Run your evaluator on these traces. You now have a set of Pass/Fail results with critiques.

This is where the flywheel connects to production monitoring: sample live traces regularly and run your evaluators on them to track quality over time.

If you have been writing critiques during manual labeling, you have already been practicing open coding. This section formalizes and scales that process using your automated evaluators’ output on new traces.

For each failed trace, write a short, informal note describing what went wrong. Do this in your own words, free-form. These are called “open codes” in qualitative research. Keep writing until patterns emerge. Examples include “hallucinated product feature,” “wrong tool call,” “missed escalation,” “bad formatting,” or “wrong tone.”

Once you have enough open codes, use an LLM to help group them into higher-level categories (axial codes). For example, individual notes about “ignored user’s refund request,” “didn’t acknowledge frustration,” and “transferred too late” might cluster into a category called “human handoff issues.” Review and edit these categories yourself. Make labels specific and actionable. Merge or split until they feel right. Add a “none of the above” category so the LLM can signal gaps and help you discover new patterns. Ultimately, identify the most frequent categories. This tells you where to focus.

Not all failure categories are equally important. Your overall pass rate can be misleading. As Jason Liu warns, aggregate metrics lie [7]. You need to look at each cluster individually.

Rank each error cluster using a 2×2 matrix with two dimensions:

• Frequency (Volume): How often does this error occur?

• Severity (Impact): How bad is this error when it does occur?

This gives you four quadrants:

• High frequency + High severity: Your top priority. Fix these immediately. These are the errors that happen often and hurt the most.

• High frequency + Low severity: Important to address, but less urgent. They are annoying but not critical.

• Low frequency + High severity: Monitor closely. They don’t happen often, but when they do, the consequences are serious (e.g., safety issues, data leaks).

• Low frequency + Low severity: Deprioritize. These can wait.

For a more nuanced prioritization, compute: Priority = Frequency × Severity × Business Value. A low-frequency error might jump to the top of the list if it directly impacts revenue or user safety. For example, a “hallucinated pricing” error might only happen 5% of the time, but its business impact is critical. This is far more important than a 30% “formatting” error that merely annoys users. Context matters: let business value break ties and override pure frequency counts.

This step also helps you surface problematic traces for review beyond user feedback. Your evaluators can proactively identify issues that users haven’t yet complained about. The goal is to turn vague impressions (“the app feels off”) into specific, ranked problems (“hallucination errors account for 25% of failures and have a critical business impact”).

For the highest-priority error categories, take action. Simple fixes might involve adjusting prompts, fixing tool configurations, or updating system instructions. Complex fixes might require redesigning the agent’s workflow, adding new tools, or restructuring the context. After fixing, add the previously failing traces to your dataset as regression test cases. Run your evaluators again to verify the fix worked and didn’t break anything else.

The first round of error analysis is a one-time investment of about 3–4 days. After the initial setup, 30 minutes per week is enough to review the latest failures and top categories, fix the easiest high-impact issues, and add or refine an evaluator only for stubborn, important problems. Re-run a full error analysis when you see a spike in failure rates, when user feedback reveals a new class of issues, or when your evaluators start feeling stale.

Over time, the flywheel converges. You fix the biggest problems first, your failure rates drop, and the remaining issues become smaller and less frequent. How often you need to run the flywheel depends on your online signals: are users satisfied, or are there anomalies?

🔎 Based on our example, let’s suppose you run your LLM judge on 200 new production traces from the email/Slack assistant. It flags 60 failures. You write open codes for each. These are quick notes like “replied to phishing link,” “mocked colleague’s achievement,” “leaked Annual Recurring Revenue (ARR) to external contact,” or “no reply to urgent CEO request.” Then you use an LLM to cluster these into axial codes:

• Tone & Professionalism Issues (18 failures): hostile replies, dismissive responses, overly casual tone with executives.

• Security Awareness Failures (14 failures): engaging with phishing, falling for CEO impersonation scams, clicking malicious links.

• Information Leaks (10 failures): sharing confidential financials, disclosing unreleased product plans, revealing salary data to strangers.

• Missing/No Response (9 failures): ignoring urgent requests, leaving teammates blocked, not confirming time-sensitive deadlines.

You rank them: Security Awareness Failures are high-frequency and high-severity (financial and safety risk), so they are the top priority. Tone issues are high-frequency but lower-severity. Information Leaks are lower-frequency but high-severity. You fix the top cluster first, then move down the list [7].

Sometimes your error analysis reveals that a single, generic evaluator isn’t enough. Different types of errors need different evaluators.

Create Specialized Evaluators

Real-world AI apps don’t do just one thing. A customer support bot handles refunds, shipping questions, account issues, escalation, and more. Each of these capabilities has its own definition of “good” and its own failure modes.

A single, generic binary evaluator (Pass/Fail on “overall quality”). Like the one you built in the previous section. It can catch broad issues but will miss category-specific problems. When your error analysis reveals distinct clusters of failures that require different evaluation criteria, it is time to create specialized evaluators.

Up to this point, your LLM judge has been a generic binary evaluator. It checks whether a trace is generally “good” or “bad.” But the error clusters the previous section might reveal that you need:

• A judge specifically for “hallucination” (did the system make up information not in the context?)

• A judge specifically for “escalation quality” (did the system hand off to a human at the right time, with the right context?)

• A judge specifically for “tone” (was the system’s tone appropriate for a frustrated user?)

Keep each specialized evaluator tightly scoped: Each judge should evaluate only one specific failure mode. Do not build a single judge who tries to assess everything at once. This keeps each judge simple, debuggable, and accurate.

Each specialized evaluator has its own rubric, a few-shot set of examples (drawn from the critiques in your labeled dataset for that specific failure mode), and its own pass/fail definition. Just like the generic judge, the effectiveness remains in the few-shot examples. But now those examples are curated specifically for one failure mode, making the judge even more precise.

Only create a new evaluator when error analysis shows a persistent, high-impact failure category that your generic evaluator can’t reliably detect. Do not create evaluators speculatively. Let the data tell you what you need. Pick 4–7 high-value failure modes that happen often enough to matter and don’t get reliably fixed by a simple prompt change [8]. Start there and add more only when error analysis demands it.

🔎 In our email assistant example, your generic evaluator catches broad failures, but the error analysis showed that “Security Awareness Failures” keep recurring even after prompt fixes. The agent still occasionally engages with sophisticated phishing attempts. You create a specialized evaluator scoped to just this failure mode: “Did the agent reply to a message that shows signs of phishing, scam, or social engineering?”

It uses few-shot examples drawn specifically from your security-related failures (the “Prince Makumba” scam reply, the fake Google alert engagement, the CEO impersonation wire transfer).

Separately, you notice “Information Leaks” also persist. So you build a second evaluator: “Did the agent disclose confidential company information (financials, roadmap, acquisitions, salaries) to an external or unverified contact?” Each evaluator has its own few-shot examples, its own pass/fail definition, and checks exactly one thing.

Let’s see all of this in action with a hands-on demo.

Demo

AlignEval is an open-source tool created by Eugene Yan that embodies the error analysis framework we have been discussing. Its tagline: “Making Evals Easy, Fun, and Semi-Automated”. It provides a streamlined interface for the exact workflow this article teaches: look at your data, label it, evaluate outputs, and optimize your evaluators.

Here is an end-to-end demo of how to label your dataset and build a binary LLM Judge with it ↓

The tool is open source and available at aligneval.com, with the source code on GitHub (eugeneyan/align-app). You can try it for free with your own data or use the prompt below to quickly generate a CSV similar to the one from the demo:

I want you to generate a CSV file with the following characteristics:
"""
* The CSV file must include the following columns:
   * id: Unique identifier for each row
   * input: Context used to generate output
   * output: Generated text to be evaluated
   * label: Ground truth (values optional but counts towards XP)
   * explanation: A one-sentence explanation on why we labeled the row as 0 (PASS) or 1 (FAIL)
* 🚨 The label column only accepts binary labels, either 0 or 1.
   * 0: Output PASSES your evaluation
   * 1: Output FAILS your evaluation
"""
that contains 100 rows

The goal of the CSV file is to implement a dataset to build an LLM Judge evaluator. 

We want to create some mock, synthetic data to conceptually show how labeling, evaluating and optimizing the LLM judge would look like, based on this tool: https://aligneval.com/

Let's say that we collected data from a vertical assistant agent specialized in answering work emails and Slack messages. Thus, create 100 scenarios based on these dimensions:
* feature: email/slack
* scenario: executive, manager, colleague, spam email, phishing email, friend (as an exception)
* label: success/failure of properly answering the message

Where the input is a single email or Slack message or an email or Slack thread, but the output will ALWAYS be just the generated reply, whether it's email or Slack.

Make the labels a 50%/50% split between passes and fails.

Also, note that NO REPLY is an expected behavior for SPAM and phishing emails. Also, for non-essential emails or toxic or slack messages.

We used Claude Opus 4.6 within the Claude app to generate it.

Next Steps

Building an evals dataset is not a one-time task. It is a continuous flywheel driven by the error analysis framework. Start small, let the data guide you, and grow your dataset and evaluators iteratively.

The full flywheel is: create dataset → label with pass/fail and critiques → fix errors → build evaluators iteratively → run error analysis → create specialized evaluators → repeat. The key principle is: “Put the data and metrics first, not preset criteria or LLMs.”

Now that you know how to build and grow an evals dataset from real data, the next article will show you how to generate synthetic test examples, extremely useful before going to production or when you don't have enough users.

Also, remember that this article is part of a 7-piece series on AI Evals & Observability. Here is what’s ahead:

1. Integrating AI Evals Into Your AI App

2. Build an AI Evals Dataset from Scratch ← You just finished this one

3. Generate Synthetic Datasets for AI Evals

4. How to Design Evaluators

5. How to Evaluate the Evaluator

6. RAG Evaluation: The Only 6 Metrics You Need

7. Lessons from 6 Months of Evals on a Production AI Companion

See you next Tuesday.

Paul Iusztin

What’s your opinion? Do you agree, disagree, or is there something I missed?

Leave a comment

Enjoyed the article? The most sincere compliment is to share our work.

Share

Go Deeper

Go from zero to production-grade AI agents with the Agentic AI Engineering self-paced course. Built in partnership with Towards AI.

Across 34 lessons (articles, videos, and a lot of code), you’ll design, build, evaluate, and deploy production-grade AI agents end to end. By the final lesson, you’ll have built a multi-agent system and a capstone project where you apply everything you’ve learned on your own.

Three portfolio projects and a certificate to showcase in interviews. Plus a Discord community where you have direct access to other industry experts and me.

Rated 4.9/5 ⭐️ by 290+ early students — “Every AI Engineer needs a course like this.”

Learn more

Not ready to commit? We also prepared a free 6-day email course to reveal the 6 critical mistakes that silently destroy agentic systems. Get the free email course.

Thanks again to Opik for sponsoring the series and keeping it free!

If you want to monitor, evaluate and optimize your AI workflows and agents:

Try Opik for free

References

1. Husain, H., & Shankar, S. (2024, January 29). Evals Are Not All You Need. O’Reilly Radar. https://www.oreilly.com/radar/evals-are-not-all-you-need/

2. Husain, H. (2024, May 6). Why is error analysis so important in LLM evals and how is it performed?. Hamel’s Blog. https://hamel.dev/blog/posts/evals-faq/why-is-error-analysis-so-important-in-llm-evals-and-how-is-it-performed.html

3. Anthropic. (n.d.). Demystifying Evals for AI Agents. Anthropic. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

4. Lenny’s Podcast. (2024, June 16). Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar. YouTube. https://www.youtube.com/watch?v=BsWxPI9UM4c

5. Husain, H. (2024, May 14). Building Eval Systems That Improve. Lenny’s Newsletter. https://www.lennysnewsletter.com/p/building-eval-systems-that-improve

6. Iusztin, P. (2025, February 11). The 5-Star Lie: You Are Doing AI Evals Wrong. Decoding AI Magazine. https://www.decodingai.com/p/the-5-star-lie-you-are-doing-ai-evals

7. Iusztin, P. (2025, February 18). The Real Battle-Tested RAG Playbook. Decoding AI Magazine. https://www.decodingai.com/p/the-real-battle-tested-rag-playbook

8. Husain, H. (2024, May 22). Using LLM-as-a-Judge For Evaluation: A Complete Guide. Hamel’s Blog. https://hamel.dev/blog/posts/llm-judge/

9. Iusztin, P. (2025, February 25). The Mirage of Generic AI Metrics. Decoding AI Magazine. https://www.decodingai.com/p/the-mirage-of-generic-ai-metrics

Images

If not otherwise stated, all images are created by the author.

🧐 Everyone says you need AI evals. Few explain how to actually build them and answer questions such as…

How do we avoid creating evals that waste our time and resources? How do we build datasets and design evaluators that matter? How do we adapt them for RAG? ...and most importantly, how do we stop “vibe checking” and leverage evals to actually track and optimize our app?

This 7-article series breaks it all down from first principles:

1. Integrating AI Evals Into Your AI App ← You are here

2. Build an AI Evals Dataset from Scratch

3. Generate Synthetic Datasets for AI Evals

4. How to Design Evaluators

5. How to Evaluate the Evaluator

6. RAG Evaluation: The Only 6 Metrics You Need

7. Lessons from 6 Months of Evals on a Production AI Companion

By the end, you’ll know how to integrate AI evals that actually track and improve the performance of your AI product. No vibe checking required!

Let’s get started.

Integrating AI Evals Into Your AI App

Understanding where AI Evals and Observability fit into the broader scheme of things can be daunting. It certainly was for me. At first, it was confusing because you can use AI evals in so many places within your application. Also, everyone seemed to have a different definition.

But it does not have to be that complicated. With this article, we want to finally connect the dots on where AI Evals fit in your AI app holistically.

But first, let’s understand WHY AI Evals are so essential.

A few months ago, I had to completely rewrite Brown, a writer agent I built as one of the capstone projects for my Agentic AI Engineering course. The first version worked but was slow and expensive. So, I redesigned the architecture from scratch.

Immediately, I hit a wall. How do I know this new version is at least as good as the old one? I had spent months fine-tuning the original and could not afford to lose that progress silently.

That is when AI evals saved me. I wrote evaluators that scored the agent on dimensions tied to our actual business requirements. With those evals, every code change generated a clear signal indicating whether I was on track. Without them, the rewrite would have been a coin flip.

You likely shipped the first version of your app. You got this far by “vibe checking” if the app works fine. Up to this point, everything is fine.

However, once you start adding new features, you realize old features break. Once you start having real users, they interact with the app in unexpected ways. If you have only 10 users, vibe checking works.

But as this scales, you get overwhelmed. You try to improve current features, and it is incredibly hard to tell if your changes have any effect. Manually managing all of this is a living hell.

💡 The solution is a structured way of measuring how well your app performs. This is known as AI Evals.

In this article, we will cover:

1. A holistic view of the AI Evals lifecycle.

2. How to use evals for optimization during development.

3. How to use evals for regression testing in CI pipelines.

4. How to monitor production quality using sampling.

5. Common misconceptions regarding guardrails and benchmarks.

6. The recommended tech stack for implementing this system.

Before digging into the article, a quick word from our sponsor, Opik. ↓

Opik: Open-Source Observability for Your Multimodal AI Agents (Sponsored)

This AI Evals & Observability series is brought to you by Opik, the LLMOps open-source platform used by Uber, Netflix, Etsy, and more.

We’re proud to partner with a tool we actually use daily across our open-source courses and real-world AI products. Why? Because it makes evaluating multimodal AI apps as easy as evaluating text ones.

AI apps are no longer just text-in, text-out. They process images, generate videos, parse PDFs, and more. Monitoring and evaluating all of that used to be painful. With Opik, it’s not. Here is why we love it:

• Trace everything — Opik renders images, videos and PDFs directly inside your traces. No more guessing what your model actually saw or generated. We use this daily, and it changed how we debug multimodal pipelines.

• Zero-friction multimodal evals — Add image URLs or upload files directly in the UI, then run LLM-as-a-Judge evaluations on them. Opik auto-detects vision-capable models (GPT-4o, Claude 3+, Gemini) and warns you if the model doesn’t support vision.

• Video generation? Traced automatically — Wrap your OpenAI client in one line, and Opik tracks the full Sora workflow: creation, polling, download, and logs the generated video as an attachment. Full visibility, minimal setup. Guide here.

Opik is fully open-source and works with custom code or most AI frameworks. You can also use the managed version for free (with 25K spans/month on their generous free tier). Learn more about evaluating multimodal traces:

Evaluate multimodal traces

↓ Now, let’s move back to the article.

The Holistic View of AI Evals

At their heart, AI Evals are systematic data analytics on your LLM application. You look at the data flowing through your app, create metrics for what matters, and use those metrics to measure what is happening. This allows you to iterate, experiment, and improve with confidence rather than guess [1], [2].

Without evals, every prompt change is a coin flip. With them, you have a concrete feedback signal to iterate against [3].

In this article, we will focus on the where, when, why, and what. In future articles from the series, we will focus on the how. There are three core scenarios where AI Evals play a central role:

1. Optimization: During development, we use evals to measure and optimize current or new features.

2. Regression: During development, when changing the code, we use evals to ensure our changes do not break previous features. This is conceptually similar to classic software tests.

3. Production Monitoring: In production, we use evals to detect potential performance issues caused by unexpected user behavior or drift.

Beyond these three, two complementary signals round out the picture. We touch on them briefly later, but they are not the focus of this article:

4. User Feedback: These are direct signals from users that bypass all our predefined datasets and evaluation strategies. They are the most valuable signal you can get.

5. A/B Testing: This ensures new code changes perform as expected, tested on real user behavior rather than predefined datasets.

This is illustrated in Image 1, which maps these concepts to the development lifecycle.

Now that we have the big picture, let’s dig deeper into each of the three core scenarios, starting with optimization.

Using Evals for Optimization

The first major use case for AI Evals is optimizing your application on a specific feature during development.

To keep costs under control and run tests multiple times during development, we run the AI evals only on a subset of the dataset that targets the feature we want to optimize. This is usually triggered manually by the developer while testing new code.

This makes development guided by concrete numbers that can be measured against a baseline, rather than just vibe-checking.

Suppose you have a customer support bot and you want to improve how it handles refund requests. You do not run your evals on the entire dataset, which may also cover shipping questions, account issues, and technical support. Instead, you filter your evals dataset down to just the refund-related examples and iterate on that subset.

You tweak the prompt, run the evals, check if your “refund accuracy” metric improved compared to the baseline, tweak again, and repeat. This keeps each cycle fast and cheap, so you can iterate multiple times in a single session [4].

But what happens when your optimization work accidentally breaks something that was already working? That is where regression testing comes in.

Using Evals for Regression

Regression testing is used to catch potential errors introduced by your new changes before they reach production. Unlike optimization, which focuses on a subset, regression testing runs on the whole evaluation dataset.

This typically happens within the CI pipeline. Because running AI Evals on the entire dataset is costly (some evaluators rely on LLM calls to grade outputs), we try to avoid running them on every single commit, as we do with standard software tests. A common approach is to run this suite when you think you are “done” with your feature, right before merging the feature branch, to ensure you are not introducing new bugs.

Continuing the customer support bot example: after you optimized refund handling during the optimization phase, you now want to merge your changes. Before merging, your CI pipeline runs the full eval suite. This includes refunds, shipping questions, account issues, and escalation scenarios.

This catches regressions. Maybe your prompt change for refunds accidentally made the bot worse at routing shipping complaints to the right team. If any metric drops below the baseline threshold, the pipeline fails and blocks the merge until you fix it. This is similar to how Anthropic’s Claude Code team and Bolt’s AI team run separate eval suites for quality benchmarking and regression testing on each change [1].

Optimization and regression testing happen during development, but what about after you deploy? Let’s look at how AI Evals work in production.

Using Evals for Production Monitoring

Production monitoring is similar to regression testing, but instead of running it offline on your AI Evals dataset, we aim to catch issues in the production environment using live traces tracked by your LLMOps platform. The final scope is to identify potential pitfalls in our system and generate alarms or warnings.

To keep costs under control, we apply smart live sampling techniques within your LLMOps platform (e.g., Opik). You rarely want to evaluate 100% of production traffic with an LLM judge. Instead, you use:

• Random sampling: Evaluate a fixed percentage of all traces (e.g., 5-10%) to get an unbiased baseline of overall quality.

• Stratified sampling: Divide traces into meaningful subgroups (by feature, user segment, query type) and sample from each proportionally, ensuring no critical category is overlooked.

• Signal-based sampling: Prioritize traces that show suspicious signals. These include long exchanges, repeated questions, user frustration indicators (thumbs-down, drop-offs from your user feedback pipe), low confidence scores, or anomalous latency/cost spikes. These are the highest-value traces to review.

You should run these as soon as practical. This can be near real-time or on a batch schedule (e.g., nightly), depending on risk tolerance and cost.

Suppose your customer support bot is live with thousands of conversations per day. Even with a 95% success rate, that still amounts to dozens of failures daily. That is far too many to review manually. Your LLMOps platform samples a percentage of live traces and automatically runs evaluators on them. For instance, you might check if the bot hallucinated a return policy that does not exist or if it escalated appropriately when the user was frustrated.

These evaluators flag problematic traces and feed dashboards that track failure rates over time. When you see a spike, you catch it within hours instead of waiting for user complaints to pile up. Perhaps a new product launch causes questions the bot wasn’t trained on [5].

Beyond evaluators, you have two complementary signals. User Feedback (thumbs-up/down, comments) is the most valuable quality signal because it reflects real satisfaction, not proxy metrics. A/B Testing validates that improvements measured offline actually hold up under real user behavior by routing traffic to different variants.

Now that we understand where and when to run AI Evals, let’s clear up some common misconceptions that trip up most teams.

Looking at Common Misconceptions

There are three major areas where terminology gets confusing: guardrails, benchmarks, and software tests.

Guardrails vs. Evaluators

Guardrails run on the inputs and outputs of the LLM or other components of your AI app. These should be very fast to avoid adding extra latency. Their role is to flag inputs/outputs as valid or not, or mask sensitive data. Evaluators, on the other hand, are used to compute metrics on your AI app components.

While you can use evaluators as guardrails if they detect adherence to business outcomes, this is the exception, not the rule. Evaluators are usually designed for accuracy rather than low latency.

For example, a guardrail in your customer support bot checks every user message in real time. If the user pastes a credit card number, the guardrail masks it instantly. On the output side, another guardrail blocks any response that promises a refund above a certain threshold. These must run in milliseconds.

An evaluator runs after the fact (or offline on sampled traces). It measures whether the bot’s refund responses are actually accurate, helpful, and aligned with company policy. The evaluator can take seconds or even minutes per trace because it is not in the user’s critical path.

App Evaluators vs. LLM Evaluators

App Evaluators measure your whole app as a unit (LLM calls + everything around them). They focus on ensuring the performance of your business use case.

LLM Evaluators measure only the performance of the LLM itself, rarely considering your business use case. Popular benchmarks like the LLM arena evaluate only the LLM in isolation. That is why benchmarks are deceiving and should never be your only criterion when picking an LLM. They are often a marketing strategy for foundational model companies.

Examples like Chatbot Arena (LMSYS) or MMLU tell you which LLM is “generally smarter.” But they say nothing about whether that LLM will handle your specific refund policy correctly, escalate frustrated users at the right moment, or respect your company’s tone of voice. You need app-level evaluators grounded in your business use case, not generic benchmark scores [4].

Evaluator vs. Classic Software Tests

When running evaluators as regression tests, they are conceptually similar to classic software tests. Their purpose is to ensure everything still works after you change the code. However, the implementation is vastly different.

Classic software tests are deterministic. For a given state of the database and a given input, you almost always get the same output. It is also much cheaper and easier to run because the code itself is cheap to run, and the outputs are structured and easy to validate.

AI Evaluators must assess the quality of LLM calls operating in a non-deterministic environment, often with unstructured data. Instead of writing unit and integration test cases, AI evals cases are operated as eval datasets, reflecting the AI-centric approach.

With a clear mental model of what AI Evals are and aren’t, let’s look at the tools you need to put this into practice.

So What Is the Tech Stack?

To run AI Evals effectively, you need two core tool families: an annotation tool and an LLMOps platform.

First, should you build a custom annotation tool or use an off-the-shelf tool? Since your data is always custom, we recommend building the annotation tool from scratch. With current AI coding tools such as Claude Code, Cursor, or Lovable, doing this is extremely easy. You want to make annotation effortless, adding zero resistance to how your data is displayed. As your data is custom, no pre-defined tool can do that perfectly for you. Most LLMOps platforms will have a feature around this, but a custom lightweight tool often wins on speed and usability.

Second, you need an LLMOps platform. Our favorite vendor is Opik. It is what we recommend and use in all our products. It is open source, constantly updated with new features, works out of the box with popular LLM APIs and AI Frameworks, and offers a generous freemium plan.

Other strong options include LangSmith, which is best for the LangChain ecosystem, and LangFuse, another solid open-source alternative. We have also heard good things about Braintrust and Arize.

The reality is that most of the time, you should pick the best tool for your current setup. We use Opik, but most of these tools have overlapping features. Choose the one that best fits your ecosystem and connections.

Next Steps

AI Evals are not optional. They are a structured, repeatable way to ensure your AI app actually works, both during development and in production.

Now that we understand the where, when, why, and what of AI Evals, the next article will focus on the how. Specifically, we will dive into how to gradually build an evals dataset.

Also, remember that this article is part of a 7-piece series on AI Evals & Observability. Here is what’s ahead:

1. Integrating AI Evals Into Your AI App ← You just finished this one

2. Build an AI Evals Dataset from Scratch

3. Generate Synthetic Datasets for AI Evals

4. How to Design Evaluators

5. How to Evaluate the Evaluator

6. RAG Evaluation: The Only 6 Metrics You Need

7. Lessons from 6 Months of Evals on a Production AI Companion

See you next Tuesday.

Paul Iusztin

What’s your opinion? Do you agree, disagree, or is there something I missed?

Leave a comment

Enjoyed the article? The most sincere compliment is to share our work.

Share

Go Deeper

Go from zero to production-grade AI agents with the Agentic AI Engineering self-paced course. Built in partnership with Towards AI.

Across 34 lessons (articles, videos, and a lot of code), you’ll design, build, evaluate, and deploy production-grade AI agents end to end. By the final lesson, you’ll have built a multi-agent system and a capstone project where you apply everything you’ve learned on your own.

Three portfolio projects and a certificate to showcase in interviews. Plus a Discord community where you have direct access to other industry experts and me.

Rated 4.9/5 ⭐️ by 290+ early students — “Every AI Engineer needs a course like this.”

Learn more

Not ready to commit? We also prepared a free 6-day email course to reveal the 6 critical mistakes that silently destroy agentic systems. Get the free email course.

Thanks again to Opik for sponsoring the series and keeping it free!

If you want to monitor, evaluate and optimize your AI workflows and agents:

Try Opik for free

References

1. Anthropic. (n.d.). Demystifying evals for AI agents. Anthropic. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

2. Husain, H. (n.d.). LLM Evals: Everything You Need to Know (Evals FAQ). Hamel’s Blog. https://hamel.dev/blog/posts/evals-faq/

3. Lenny’s Podcast. (n.d.). Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar. YouTube. link

4. Reganti, A. N., & Badam, K. (2025, January 28). Evals Are NOT All You Need. O’Reilly. https://www.oreilly.com/radar/evals-are-not-all-you-need/

5. Habib, R. (2024, March 14). Why Your AI Product Needs Evals with Hamel Husain. Humanloop Blog. https://humanloop.com/blog/why-your-product-needs-evals

Images

If not otherwise stated, all images are created by the author.

🧐 Everyone says you need AI evals. Few explain how to actually build them and answer questions such as…

How do we avoid creating evals that waste our time and resources? How do we build datasets and design evaluators that matter? How do we adapt them for RAG? ...and most importantly, how do we stop “vibe checking” and leverage evals to actually track and optimize our app?

This 7-article series breaks it all down from first principles:

1. Integrating AI Evals Into Your AI App

2. Build an AI Evals Dataset from Scratch

3. Generate Synthetic Datasets for AI Evals

4. How to Design Evaluators

5. How to Evaluate the Evaluator

6. RAG Evaluation: The Only 6 Metrics You Need

7. Lessons from 6 Months of Evals on a Production AI Companion ← You are here

By the end, you’ll know how to integrate AI evals that actually track and improve the performance of your AI product. No vibe checking required!

Let’s get started.

Lessons from 6 Months of Evals on a Production AI Companion

Paul: Today, the stage belongs to Alejandro Aboy, Senior Data Engineer at Workpath. Alejandro owns the entire data stack and is the architect behind the Workpath AI Companion.

When he isn’t shipping, he’s sharing engineering deep-dives at his publication, The Pipe & The Line.

Enough chitchat. Let’s get into it 👀 ↓

Alejandro: Over the last year, I’ve been working on different agent projects, but one took most of the attention.

We are talking about the main AI feature of the company SaaS, the companion that guides you through everything you can do and can also do it for you.

It can call around 50 different tools, do RAG searches based on documentation and also save workflows in memory so you don’t have to start from scratch. It can even get contextual messages from the frontend to know what you see when you see it on the platform.

As you can imagine, there’s a lot it can happen when you give too much power to Agents. These are some of the things I noticed:

• Agent complies with formatting but hallucinates documentation links after fake searching the docs

• Agent infers the wrong user information and makes the wrong tool calls or argues no information can be found

• Agent suggests follow up with actions that are outside scope, like running tool calls that don’t exist

There are lots of Agent demos out there, and 2026 won’t stop the hype probably, but we barely discuss these nuances and how to address them in production environments to keep improving our Agents with awareness of what’s happening after we do a prompt change or add a new tool.

To address that, here is what we will learn in this article:

• Problems with AI Observability

  • Falling For Classic Metrics Or Trying To Define What’s “Good” or “Bad”

  • Not Going Through Manual Annotations

  • Not Treating Your AI Agents As A Data Product

• Implementing AI Observability

  • Opik Overview

  • Opik MCP Server

  • Figuring Out Evaluation Criteria

  • Refining Evaluation Criteria

  • Running Annotation Sessions

  • Making Sense Of Annotated Feedback

• Backstory Of This Framework

But first, a quick word from our sponsor, Opik ↓

Opik: Open-Source LLMOps Platform (Sponsored)

This AI Evals & Observability series is brought to you by Opik, the LLMOps open-source platform used by Uber, Etsy, Netflix, and more.

But most importantly, we are incredibly grateful to be supported by a tool that we personally love and keep returning to for all our open-source courses and real-world AI products. Why? Because it makes escaping the PoC purgatory possible!

Here is how Opik helps us ship AI workflows and agents to production:

• We see everything - Visualize complete traces of LLM calls, including costs and latency breakdowns at each reasoning step.

• Easily optimize our system - Measure our performance using custom LLM judges, run experiments, compare results and pick the best configuration.

• Catch issues quickly - Plug in the LLM Judge metrics into production traces and receive on-demand alarms.

• Stop manual prompt engineering - Their prompt versioning and optimization features allow us to track and improve our system automatically. The future of AutoAI.

Opik is fully open-source and works with custom code or most AI frameworks. You can also use the managed version for free (w/ 25K spans/month on their generous free tier).

Try Opik for free

↓ Now, let’s move back to the article.

Problems With AI Observability

As mentioned, there are many things you don’t see on how you Agent is working, and depending on usage and scale, you are likely to miss most of them.

The problem won’t be gone by reviewing everything manually or finding the perfect sample to evaluate.

There are some problems I learned about while implementing it at work. These have been mentioned by great experts on the topic, but I summarised how I experienced them like this:

Falling for Classic Metrics or Trying to Define What’s “Good” or “Bad”

Tools like Opik or Langfuse come by usual metrics out of the box Usual metrics cover Hallucination, AnswerRelevance, ContextPrecision and ContextRecall.

You can configure them and start evaluating right away, but we you get that the hallucination score is 1, then what?

You need to find what to evaluate to improve your use case, otherwise you can’t really improve it.

By this point, if you used LLMs for anything, you might have learn about determinism and its nuances. You can run the same thing over and over and get different outputs.

Same with evaluations; you need to find a standard way of evaluating without struggling with ambiguity every single time.

That’s why trying to evaluate nuances and all flavours of a metric might cause confusion.

You need to keep it lean, the best test is asking different people what they think and if they answer the same, you got a good metric.

For example, I wanted to evaluate if the agent was hallucinating documentation links. My evaluation task was: Check if search_knowledge tool was called and verify if the URL in the output matches the tool output the agent used. That’s binary and can’t be misinterpreted even if you run the LLM As Judge multiple times.

Not Going Through Manual Annotations

Unfortunately you can’t get away with murder without reviewing Agent responses manually.

If you automate 100% of those reviews with LLM as Judge you are likely to miss the substance because LLMs might hide interesting finding behind isolated prompts that could be phrased badly.

For example, all my metric scores were doing mostly fine, but when I started seeing some conversations manually I noticed the agent suggesting things it can do at all, or answering things outside scope. That surface more evaluation opportunities, but also more things to add to the backlog that we could not see otherwise.

With all the MCP servers out there, you can easily cluster all the manual feedback you collected and prepare documents for further analysis, so there’s no excuse for “who will review all these manual comments later”?

Not Treating Your AI Agents as a Data Product

The worst mistake someone can do is just evaluating an AI Agent and not take anything out of it besides good or bad metric scores.

Each conversation its writing an invisible roadmap you need to materialise.

Whether if its a JIRA ticket to address a bug, or a new feature development plan for a brand new Agent capability.

I personally developed a lot of Product Analytics use cases to cover for internal analytics done by CS teams, which allowed a lot of reverse engineering to discover new use cases.

If you just create synthetic data to evaluate a prompt and leave observability at that point, you are missing out on real production data that can scream improvements right to your face.

So shift your mindset and own the process to define the success of each one of your AI Agent projects ;)

Recommended:

• Decoding AI Magazine - The 5-Star Lie: You Are Doing AI Evals Wrong

• Decoding AI Magazine - The Mirage of Generic AI Metrics

Implementing AI Observability

In the upcoming sections of this article, we will talk about Opik, but these fundamentals are relevant for any other observability tool.

It’s quite common in Software Engineering to use tools such as Sentry for error logging.

Opik Overview

Tools like Opik follow the same principles: traces & spans.

You trace every single LLM interaction (trace), whether is an AI Agent or just a simple LLM call you want to audit.

You can break it down to see every thing that got used along the way (span), which translate into agent tools, such as retrieving documents with RAG.

All traces are grouped into threads, which equal to a chat or conversation putting messages together. When reviewing LLMs, it’s relevant to look at the thread level to get a grasp of how it worked considering the full context.

Beyond the trace <> span pair we talked about, with Opik you can:

• Do prompt versioning, which comes in handy to associate metric scores evolving after a particular prompt version.

• Connect your AI provider (OpenAI, Anthropic, Azure, etc) in order to run Online Evaluations every time a new trace comes in, so you get fresh metrics on how your Agent is working on production.

• Use its SDK & REST API to scale any of its processes, from trace processing to running on demand evaluations using LLMs.

I am just scratching the surface here, you can do a lot more and I encourage you to take a look!

Recommended:

• Decoding AI Magazine - Observability for RAG Agents

Opik MCP Server

Using the MCP Server is optional as any other MCP, but I found it super powerful to do things that would have take me weeks or even months.

I found ways of leveraging to coming up with metric ideas based on my prompts and traces data, also picking up comments to quickly write documentation to review for potential product roadmaps.

You can get creative pretty easily and I invite you to try it.

Take a look on Github - opik-mcp to get started.

For the upcoming sections we will be using some slash commands that wrap some workflows to enhance the annotation process to make it more streamlined and enriched.

You can find them here: Opik MCP Commands

⚠️ Disclaimer: Opik MCP tools return a bunch of data. You are likely to see this message “⚠ Large MCP response (~10.6k tokens), this can fill up context quickly”.

The CLAUDE.md instructions are meant to tell the LLM how to use tool calling to sample data gradually to avoid filling up the context quite quickly. The ideal workflow is too sample some threads and keep going as needed and saving checkpoints of the analysis to .md files for future reference.

Figuring Out Evaluation Criteria

On Opik, you can run evaluations on the trace or thread levels, this means you can evaluate the single turn (user message > agent message) or multi turn interactions.

But first we need to understand what our Agent is actually doing.

If we don’t know the use cases its covering, its really hard to know how to evaluate it accordingly.

Note: All these commands are useful with real production traces, not synthetic data. You want to know what the agent is being used for, not what you want it to be used for.

To achieve this, you need a prompt created on Opik so you can use Claude Code or Cursor to reverse engineer it to figure what are the best evaluation criteria principles to cluster.

When you run opik-eval you get something like this:

Now that you have your prompt fully analysed, you will proceed to break it down on how to evaluate it based on usage.

Then it will propose metrics for Trace and Thread levels, looking like this:

The commands are expected to propose around 5 metrics for trace evaluation, but you can optimise it to your sweet spot depending on how many evaluation metrics you want to handle simultaneously.

Also, automating evaluation on the thread level can be tricky, since its hard to evaluate without too much ambiguity. Based on my experience, I will only evaluate threads manually since it’s where most of the subjective value lives.

You can iterate and use the template as much as you can to come up with the most suitable evaluation angles for your project. When you are ready, you can go to configure your Online Evaluations. This means deciding on:

• scope: Trace or thread.

• model: What you configured in your AI provider.

• prompt type: In here you will paste the proposed output of the Claude Commands, but you can use other default ones Opik provides, such as Hallucination or Answer Relevance.

You can add new variables, such as context (seen in the example output) by writing {{context}} in the prompt.

When you click on “Select a key from recent trace”, it will display, based on your tracked traces, all the possibilities depending on your framework, since the keys listed will depend on what you use, since Langchain will differ from OpenAI SDK or other frameworks.

Later on, you will define the score definition with a name, a short description on what the score is about and its metric type (boolean or categorical).

Lastly, you can configure how many traces you want to evaluate, since you might want to sample a representative amount instead of just evaluating everything. Filters on different trace dimensions are allowed, which can help you clean up and avoid evaluation pollution.

Based on all my experiments, the ideal evaluation metric focus about ONE thing in the prompt and ONE score definition. Metric type can be flexible as long as you avoid subjective outcomes.

Going back to the Claude Command output, you will get a summary of what cannot be evaluated based on the trace information it has found, this is helpful for extra brainstorming on what to actually evaluate:

Refining Evaluation Criteria

Now that you have your online evaluations defined and active, your traces will start getting some evaluated outputs.

Versioning Online Evaluations is not possible, so you need to document somewhere that you did that change.

For example, my “Response Format Compliance” evaluates if the agent is following the prompt formatting standards properly.

When I saw this pattern, I check 10-15 traces randomly and realised the condition for scoring 1 was “to match at least 2 of 5 formatting standards”, so I refined it to be “to match ALL”.

If after some days the score is fixed to 1 and after further review the score makes sense, I would conclude this metric its either saturated or not relevant if its always going to be ok.

I like to go through this manually since Opik dashboards are quite quick to scan through and know where to deep dive, but you can iterate through this within a conversation with your AI IDE.

Once you have the rationale for these changes, I recommend leaving some notes on Google Docs, Notion or Confluence pages to track some sort of version control over time.

Running Annotation Sessions

Annotation Sessions can be painful and boring.

After manually reviewing 500 conversations, I decided to come up with a slash command to setup a mindset on how the last days would look like before I even jumped to review.

This helped me a lot because I was always to walk through the UI and filter some patterns to better conduct my reviews.

Note that you need to have Feedback Definitions in place, which are different from Online Evaluations

While Online Evaluations are triggered as traces or threads arrive, Feedback Definitions are used when running manual annotation to give feedback on how the Agents are performing.

Running opik-weekly gets you an output like this one:

You have a first glance on the feedback score overview, any latency outliers and token usage to further deep dive.

Then you get some anomaly analysis on thread level with some WoW (Week Over Week) comparisons:

With the truncated messages you can go to Opik Threads or Trace views and filter by message so its quite quick to analyse some anomalies or outliers you found interesting.

Making Sense of Annotated Feedback

You can get a quick start before jumping into annotations.

Running opik-annotation-review gets you an output like this one:

This can give you an overview of what’s been commented last time, also useful to connect with another MCP such as Atlassian and create tickets on JIRA or update documents on Confluence.

It’s also helpful to quickly check backlogs to validate if the most concerning highlights are considered for further developments.

Backstory of This Framework

It’s been almost a year since I implemented the first agent projects at work.

I am using PostgreSQL for memory and session tracking for conversation history, so it's quite easy to run some wrapped SQL queries in order to get an idea of usage for Product Analytics use cases.

I did that… for around 6 months.

But just saying that we have 100 messages a day or 20 new users every week, I wasn’t getting any input on how to improve the agents.

Motivated by frustration, I started researching and experimenting with AI Observability.

Read a lot, followed a lot of projects implementing tools like Opik or Langfuse.

Saw perfect evaluation scores on top of synthetic datasets, tried sample projects with really nice constructed conversation threads.

And then I started hitting walls:

• I built scripts (meant to run locally) to run LLM As Judge evaluations at scale.

• I did Prompt Versioning manually, copy pasting my changed prompt.

• Wondered how to create golden datasets while I had a bunch of real data to actually reverse engineer the process.

• Guessing what to evaluate, without actually looking at how my prompts were reacting to user messages.

• Now knowing what threads to actually annotate because I had no system in place, I ended up annotating everything.

I got super frustrated because I was getting value out of it, but at the cost of being exhausted from a lot of manual back n forth.

All those tutorials out there were showing 1% of running AI Observability as an iterative process, when its a cyclic process that continuously evolves.

Then I discovered Online Evaluations, started experimented with Claude Code and Opik MCP and everything clicked.

At some point, I was lucky to go through all the manual process because I got a lot of input that helped figuring out evaluation criteria faster, but today I would approach completely different leveraging the right tools and saving a lot of time.

All these pains led me to come up with these Opik MCP Commands in Notion to:

• Make a data driven, reverse engineering process of the prompt you want to evaluate to come up with solid evaluation criteria.

• Have an annotation agenda before sitting down to review manually, so the process is streamlined and not easily sidetracked by distractions.

• Know more about what your agent does, without having to be everywhere, and getting time back to draw your roadmap treating it as a product and not just another thing you have to maintain.

Nothing about this framework is written in stone, and its meant to be super flexible. Before getting it to the Notion page I changed a bunch of times and it will keep evolving as the AI landscape gets more powerful.

Also, remember that this article is part of a 7-piece series on AI Evals & Observability. Here is the full series:

1. Integrating AI Evals Into Your AI App

2. Build an AI Evals Dataset from Scratch

3. Generate Synthetic Datasets for AI Evals

4. How to Design Evaluators

5. How to Evaluate the Evaluator

6. RAG Evaluation: The Only 6 Metrics You Need

7. Lessons from 6 Months of Evals on a Production AI Companion ← You just finished this one

‘Till next time

Alejandro Aboy

Do you want more articles like this from Alejandro?

If yes, consider subscribing to his amazing Substack, where he talks about data and AI from his real-world experience ↓

The Pipe & The Line

Hands-on guides, tools, and experiments to sharpen your Data & AI Engineering skills from someone who learned it all in the wild.

By Alejandro Aboy

What’s your opinion? Do you agree, disagree, or is there something I missed?

Leave a comment

Enjoyed the article? The most sincere compliment is to share our work.

Share

Go Deeper

Everything you learned in this article, from building evals datasets to evaluators, comes from the AI Evals & Observability module of our Agentic AI Engineering self-paced course.

Your path to agentic AI for production. Built in partnership with Towards AI.

Across 34 lessons (articles, videos, and a lot of code), you’ll design, build, evaluate, and deploy production-grade AI agents end to end. By the final lesson, you’ll have built a multi-agent system that orchestrates Nova (a deep research agent) and Brown (a full writing workflow), plus a capstone project where you apply everything on your own.

Three portfolio projects and a certificate to show off in interviews. Plus a Discord community where you have direct access to other industry experts and me.

Rated 4.9/5 ⭐️ by 190+ early students — “Every AI Engineer needs a course like this.”

Learn more

Not ready to commit? We also prepared a free 6-day email course to reveal the 6 critical mistakes that silently destroy agentic systems. Get the free email course.

Thanks again to Opik for sponsoring the series and keeping it free!

If you want to monitor, evaluate and optimize your AI workflows and agents:

Try Opik for free

Images

If not otherwise stated, all images are created by the author.

One service is down. Then another 10 follow. Restarts begin. Logs and dashboards come up side by side.

The graph looks familiar. The error rate spikes, rolls back, spikes again. Someone pastes a dashboard link into Slack. Someone else replies with a half-sentence: “Didn’t we see this two weeks ago?” No ticket is linked. No postmortem is found. The incident feels known, but undocumented.

A workaround exists. Everyone knows that much. Nobody knows why.

It lives in a shell script, wrapped in a cron job, guarded by a comment that says “DO NOT REMOVE”. The person who wrote it left two companies ago. The context left with them.

Slack then becomes an archaeological dig. You scroll past emojis, past renamed channels, past a debate that ends mid-thread. Somewhere in 2021, an engineer explains the underlying problem, but no action is taken for now.

This is how enterprise systems decay. Not through broken code, but through forgotten understanding.

And this is how production engineers start carrying all this history in their heads.

They know which alert is real. Which one needs a manual nudge. Where each dependency lies. Which rollback will make things worse. Over time, the entire system is held together not by documentation or dashboards, but by the accumulated memory of our production engineers.

This works until it doesn’t. When the wrong person is asleep. When someone leaves. When the system grows just large enough that no single mind can hold it all anymore.

In this article, I want to walk through a very concrete use case and design it together with you.

We will design a Production Engineer agent that reacts to alerts by identifying the affected services and teams, understanding how the issue propagates through the system, and surfacing the context that engineers need to act quickly.

By the end, you will have a clear understanding of how to design an agent for monitoring and reasoning about production systems, and more importantly, how to design the right kind of memory to support it.

The twist is that the agent’s real superpower isn’t reasoning—it’s GraphRAG.

Let’s unpack why.

But first, a quick word from our sponsor, Opik ↓

AI Agents Virtual Hackathon With $30,000 in Prizes (Sponsored)

Want to get motivated to build that AI agent you had in mind in the past 12 months while having fun, meeting cool people and potentially earning up to $10,000 (in cash per team)?

Hackathons are the best way to do that.

Opik is hosting a free one, together with Google DeepMind and Vercel, offering $30,000 in prizes with a single goal: building and shipping AI agents.

But wait. That sounds like a scam. Not really. The catch is that you have to put in the work during the hackathon to convince the judge that your AI app is worth the prize.

The worst that can happen? You have the chance, for free, to access:

• Expert Workshops: Learn observability, evaluation, and agent optimization from Comet’s team

• Premium Tools: Credits and support from Google, Vercel, and other partners

• Direct Mentorship: Technical support throughout the hackathon via Discord

Prizes will be awarded based on the 6 challenges below. You can win one category plus the best use of Opik, totaling $10,000.

My take? The money is just a bonus, as this is a fantastic opportunity to learn for free from industry experts while building:

• LLM judges to evaluate your custom business use case.

• Agentic RAG applications or coding agents.

• Automated prompt tuning loops and guardrails.

Or whatever AI agent moves and motivates you!

Who Should Join?

Developers with:

• Basic understanding of LLMs and AI agents

• Experience building software applications

• Python or TypeScript knowledge

Sounds like this is for you? Then register here:

Apply to Hackathon

Be quick! You can register at any time during the event, up to the project submission on February 9.

↓ Now, let’s move back to our article.

1. Our use case: The Production Engineer

As companies grow, their systems rarely fail in isolation. Services depend on other services, teams depend on teams, and ownership is spread across layers that are hard to see from any single place.

When something breaks upstream, the downstream impact is often unclear. Engineers are left stitching together context from dashboards, Slack threads, old announcements, and workarounds buried in Confluence.

The problem is clear. Most issues are fixable. What slows teams down is figuring out what is really happening and how far it reaches before time is lost.

This is where the Production Engineer agent comes in.

The feature we want to design is simple, but powerful. When an alert is triggered, the agent should be able to:

• understand how the failure propagates across the organization, including which services are affected and which teams are responsible

• diagnose likely causes using known dependencies and patterns from past incidents

• surface the relevant context that engineers usually spend hours reconstructing from dashboards, Slack threads, and internal documentation

Why is this important?

Because most production incidents are not slowed down by the lack of a fix. They are slowed down by the lack of clarity. Engineers need to understand what is happening, how far the issue reaches, and who needs to be involved before they can act with confidence.

The goal is to shorten the time between detection and action, especially in large enterprises where context is fragmented, and knowledge is spread all over.

To build something like this reliably, we need to carefully design the system and how information flows through it. That starts with the architecture.

2. Designing the architecture

Now, let's break down exactly how this system would work when that 02:13 pager goes off.

At that point, the goal is not just to raise an alert, but to immediately attach the right context so engineers don’t have to reconstruct it manually. The architecture is designed to do exactly that.

The interface of our Production Engineer Agent is intentionally straightforward:

• Input comes from monitoring systems via webhook API: Prometheus sends POST requests to our FastAPI endpoint, each payload including service name, error type, severity, and timestamp. The webhook fires the moment something crosses a threshold.

• Output goes to all affected teams via Slack: Structured incident reports are posted to relevant team channels, tagged with the right on-call engineers and linked to runbooks, past incidents, and related documentation.

The diagram below shows how GraphRAG and MCP wire this together:

At a high level, the system is decomposed into five components, each with a clearly defined responsibility and a well-scoped interface:

#1. Alerting System

This is a standard component in any enterprise environment. Prometheus detects threshold breaches and emits alerts, which are routed through Alertmanager. From there, alerts are delivered via webhook to our FastAPI server. This webhook is the entry point for every incident and the only way external signals enter the agent system.

#2. Agent Component

The Agent Component orchestrates the entire flow. The FastAPI server receives the alert payload and forwards it to the Agent Controller, which is responsible for handling the agent loop:

• invoking tools via the MCP Client (used to communicate with external systems),

• querying the GraphRAG component to retrieve contextual knowledge about teams, system dependencies, and ownership.

• preparing the prompt and additional context, sending it to the LLM through the LLM Gateway, which handles all direct interaction with the Gemini API.

It serves as the coordination layer between context retrieval, tool execution, and model inference.

#3. GraphRAG Component

GraphRAG serves as the agent’s long-term, structured memory. It is built on a Neo4j graph database that models services, teams, and other dependencies as a property graph, with vector embeddings attached to nodes.

The Graph Query Engine performs graph traversals to retrieve the most relevant entities and their dependencies based on the incident context. The graph itself is populated offline by a Graph Extractor, which ingests organizational data from sources such as Confluence documentation.

#4. MCP Servers

MCP servers provide real-time, external context. A global MCP router forwards requests to specialized servers:

• Confluence MCP retrieves documentation,

• GitHub MCP fetches recent code changes and release information,

• Slack MCP searches historical discussions and posts updates to support channels,

• Prometheus MCP retrieves live metrics.

This separation allows each data source to evolve independently while presenting a uniform interface to the agent.

#5. Observability

Observability is handled through Opik. Prompt Monitoring tracks the agent’s questions, tool usage, and retrieval strategies, while Trace Logging records executed queries and their latency.

Together, these signals provide visibility into how the agent reasons, what it accesses, and where time is spent.

Now that we understand each component, let’s walk through the data flow:

1. Prometheus detects a threshold breach and fires a webhook to the Alerting Manager with service name, error type, severity, and timestamp.

2. The Alerting Manager routes the alert to our FastAPI Server endpoint, which hands it to the Agent Controller.

3. The Agent Controller queries the GraphRAG Component for related context. The Graph Query Engine does a semantic search to retrieve the nodes closest (“communities”) to the query and fetches them along with their dependencies.

4. With graph context assembled, the Agent Controller sends a plan to Gemini specifying which MCP servers to call for real-time data.

5. Gemini returns the required tools. The Agent Controller invokes the Global MCP Server with the list.

6. Each requested MCP server is called: GitHub for fetching recent code changes/releases, Slack searches incident discussions, Confluence retrieves documentation, and Prometheus pulls current metrics.

7. The Agent Controller sends all context to Gemini and fresh MCP data.

8. Gemini synthesizes everything into a structured incident report with impact summary, pattern recognition, recent changes, current state, and recommended actions.

9. The report goes to the Slack MCP Server, which posts it to affected team channels.

Note: Gemini may request additional MCP tool calls as needed, meaning steps 6-11 can loop until the complete context is gathered.

And just like that, the incident is contextualized and actionable in seconds, before the engineer opens their laptop.

But this speed does not come from the agent loop alone.

What makes this possible is the memory behind it, the way organizational knowledge is stored, connected, and retrieved at the moment it is needed.

This is where GraphRAG enters the picture.

3. Graph RAG as our organization's knowledge

Before we talk about GraphRAG, we need to clarify the foundation it builds upon: the knowledge graph.

What Is a Knowledge Graph?

A knowledge graph is a structured way of representing information as a system of connected components:

• Nodes represent entities such as documents, services, concepts, teams, incidents, or decisions

• Edges represent relationships between those entities

• Properties on nodes and edges store metadata like summaries, timestamps, ownership, or importance

Unlike flat document stores or pure vector databases, a knowledge graph explicitly captures how information is connected, not just where it appears.

From a production perspective, this matters because it enables things we rely on every day:

• Tracing relationships (“what depends on this service?”)

• Aggregating information across multiple sources

• Preserving institutional knowledge beyond individual files, tickets, or dashboards

In practice, a knowledge graph becomes a shared semantic layer for the organization: a living map of how systems, documentation, decisions, and operational knowledge relate to one another.

What Is GraphRAG?

GraphRAG is Retrieval-Augmented Generation (RAG) using a Knowledge Graph.

At a high level, the difference is simple:

• Traditional RAG retrieves the most semantically similar text chunks from a vector database.

• GraphRAG retrieves connected knowledge by traversing relationships in a graph.

For us, the most useful way to think about GraphRAG is this:

GraphRAG is a set of RAG patterns where retrieval is guided by graph structure, not just similarity scores.

Each pattern depends on having the right graph representation in place.

Why GraphRAG for Organizational Knowledge?

In production environments, we rarely ask questions that can be answered by a handful of similar text snippets.

Instead, we ask questions like:

• “What do we know about this issue across teams and services?”

• “Summarize everything related to this initiative and its downstream dependencies.”

These are coverage and synthesis questions, not similarity questions.

A similarity-based retriever might return a few highly relevant chunks, but still miss:

• Entire systems

• Related incidents

• Important but differently worded documentation