This article originally appeared in Hugo’s newsletter, Vanishing Gradients, and is republished here with the authors’ permission.

Paul: Today, two of the industry’s brilliant minds own the scene.

You’ll learn from Hugo, who teaches LLM systems to engineers at Netflix and Meta, and Eleanor, a former principal engineer at Microsoft and Google who’s now building evidence-based, production-ready AI at Jimini Health and Agentic Ventures.

They’ve created this dope course on building an agentic software factory. Decoding AI community members get 25% off via the link or code FRIENDSOFPAUL at checkout, plus $500 in Modal credits. 

P.S. Take notes on this one. ✍🏼

Agentic software factories are here

Stripe’s Minions turn engineering requests into candidate pull requests. Ramp’s Inspect puts coding agents to work in the background, with tools to check their changes. Warp Factories takes work from tickets and chat through agents, checks, and human review. Agentic software factories are here.

Are you sick and tired of babysitting Codex, Claude Code, or OpenClaw?

Do you have agents that can build features, but still leave you responsible for remembering where they stopped, explaining how each feature fits the project and deciding what happens next? You make progress in each conversation, then development waits for you to return.

It doesn’t have to be this way. A software factory lets you delegate more of that process!

You agree on an outcome, give the system the context and authority it needs, and let it work while you do something else. When you return, there should be

1. a result you can inspect, with evidence of what was checked; or

2. a specific reason the work could not continue.

Your next conversation can then focus on the result and the decisions that remain.

As we’ll show you, your first software factory doesn’t need to be that complex at all: it can begin with a single agent and an automation.

What makes it a factory

The defining combination is autonomous and asynchronous execution.

• Autonomous: the agent chooses and carries out steps towards an agreed goal. It can inspect what happened, adjust its approach and continue within its authority.

• Asynchronous: that work can proceed without your ongoing attention. You can leave and return to its result.

A scheduled job can run without you. A coding agent can choose and revise an approach during an interactive session. A factory combines those capabilities, with enough context, tools and feedback to carry out the delegated work.

Software factories have been hypothesized for decades. Now that we have agents, we’re actually able to operate them. The principles apply to software work traditionally carried out by teams of humans, from bespoke tools for teams and individuals to large and complex software systems.

To illustrate these principles, we’ll use an invoice tracker throughout this article, an application we’ve needed for our own businesses. The same approach applies to research tools, publishing systems and software for teams. Suppose you run a small business and want an invoice tracker that fits how you work:

Build an app that creates invoices from my template and tracks which are unpaid, paid or overdue. Keep it local for now. Preserve the records when it restarts, and show me checks that the totals and payment statuses are correct.

You can agree on this brief with the agent, leave it to work while you do something else, and return to the app and evidence of what was checked. If a check fails, it can investigate and adjust within its agreed scope. If it encounters a product decision it cannot make, it should bring that question back to you.

You might imagine a clean separation: the factory builds the invoice tracker, then the finished app does its job. That’s one way to run it. But agents can also help operate the software they build, and that combination is useful.

The tracker can store invoices and calculate outstanding balances. An agent can use those records alongside customer correspondence to prepare follow-ups: one customer has promised to pay on Friday; another is disputing a charge. You don’t need to turn every situation into a new software feature before the agent can help. You give it the relevant context, authority and a way to check its work.

This changes what you need to build! Some behaviour belongs in code, such as calculating balances consistently. Other work can be delegated to an agent that responds to the circumstances. If a recurring need emerges, you can ask the factory to build it into the software. The factory builds tools that agents can use, and experience using those tools helps you decide what to build next.

You still decide what is worth building, who it should serve and which trade-offs matter. Agents can help you make those decisions too. The practical change is that implementation can proceed between your conversations, including running checks and responding to failures.

This makes factories relevant to people with product, business or creative expertise as well as engineers. You might know exactly how you invoice clients, record payments and chase overdue bills while having little interest in writing software. That knowledge lets you direct the factory towards software that fits your business, including the exceptions and connections that general-purpose tools handle poorly.

The software factory hierarchy of needs

Monica Rogati’s original AI hierarchy of needs described the foundations beneath successful AI work. We find that dependency framing helpful for factories too, with four different layers.

1. Harness. The environment around the model: tools, a workspace for project files, execution and feedback. It lets an agent read files, change software and run it. If a check fails, the harness must give the agent enough information to investigate what happened.

2. Personalisation. The context and capabilities for your work: project direction, rules, reference material, skills and connections to other systems. The agent needs to know what you are trying to accomplish and which existing decisions it should preserve. Permissions limit access to tools and systems; sandboxing restricts the environment where actions can run.

3. Automation. A way to start and continue work without a fresh instruction from you each time. A schedule or event can trigger a task or the pickup of ready work. It also needs direction about what qualifies as ready and where to leave the result.

4. Continual learning. A process for learning from completed and failed runs, improving the factory and carrying those changes into later work. That might change an instruction, a checking procedure or a tool the agent uses.

Automation needs a harness able to do the work, plus direction about what to do. Continual learning needs runs to inspect and a way to put improvements into use. These capabilities can develop together: a familiar harness, a short brief and a basic automation are enough to start. You can personalise further as work reveals what is missing.

Two practices run through every layer: specification and verification:

• Specification makes the intended outcome, constraints and criteria for success explicit. In our invoice brief, that includes preserving records after a restart and calculating payment statuses correctly.

• Verification checks the software and the work agents perform against those requirements.

Agents can help you refine the specification and perform the checks. Your role is to decide whether the requirements capture what you need and whether the evidence supports accepting the result. As the factory takes on more work, these practices give you a way to direct it and assess what comes back.

Let’s now see what’s up in each layer.

1. Agent harness

First, you’ll need an agent harness. If you’re using Codex, Claude Code, Hermes, or OpenClaw, you already have one. It gives the model tools to read and edit files, run your software and see what happened.

For our invoice tracker, the agent can build the app, create an invoice, record a payment and restart it to check that the invoice and payment records survived. If a paid invoice still shows as overdue, it can investigate, fix the problem and try again. You shouldn’t have to narrate every step. But if it needs a decision about how your business handles partial payments, that’s a question for you. If it lacks access or authority, the agent should stop and name the blocker.

You’ll also need somewhere for the harness to run. A cloud machine lets the agent keep working while your laptop is closed or disconnected. You can also run it on your own computer, but the machine needs to stay awake and connected for the work to continue.

2. Personalisation

Next, give the agent the context it needs to work your way. For our invoice tracker, that means your template, payment terms and conventions for recording payments. You don’t want to explain all of that again every time you start a new task.

Keep it somewhere future workers can find it:

• Rules: preserve existing invoice records; ask before changing payment-status calculations.

• References: your template, payment terms and example invoices.

• Skills: repeatable procedures for testing invoice creation, payment updates and persistence.

An instruction file such as AGENTS.md can point to this material. OpenAI’s harness-engineering account describes using a short guide that points into maintained project documentation. Give a new or scheduled worker a small task and check that it actually follows the guidance.

Be specific about when instructions apply. Eleanor once told an agent to write tests, and it wrote tests for a README. Enthusiasm wasn’t the problem 😂

You’ll also need to set permissions and sandboxing to control what the agent can access and execute. Written instructions alone don’t enforce those boundaries!

3. Automation

Now let’s give the factory a way to get started without waiting for you. A schedule or event can launch an agent with its brief, project guidance and tools. You come back to a result, evidence of what was checked or a specific blocker that needs your attention.

One agent working in the background

For our invoice tracker, a morning automation could check a saved task list and pick up the next agreed task, such as adding partial payments. You’ve decided what you want and how success will be checked. The agent gets on with it while you do something else.

There are a few practical details to get right. Don’t start the same task twice, leave an empty queue alone, and set limits on time and spending. Save progress so a later session can continue without you having to reconstruct the whole conversation.

One worker may be plenty. But perhaps you have independent tasks that could run in parallel, or you’d like another agent to review a change. That’s where coordination comes in.

Coordinated work

With several agents, everyone needs somewhere to see what’s ready, what’s underway and what’s waiting on something else. A shared task list or issue tracker can hold that information. You agree which tasks are ready; a coordinating agent or automation assigns them, supplies the relevant context and records what happens.

Suppose one task adds partial payments, another overdue follow-up emails, and a third PDF exports. Follow-ups need the correct outstanding balance, so they wait for the payment behaviour to be checked. PDF exports might proceed independently if the changes don’t overlap.

You can also split the roles. One agent builds the feature while another prepares payment scenarios to test it. A third might review the completed change.

Assign responsibility for combining the changes and verifying the whole application. Different questions call for different checks. Deterministic tests can establish whether balances are calculated correctly and payments survive a restart. Agent verification can help identify missed requirements or inconsistencies across the changes. Human judgment can assess whether the workflow fits how you actually work. You’ll often want a combination, with more than one kind of check on the same result. Anthropic’s evaluation guide describes these complementary approaches: code-based checks, model-based checks and human evaluation.

Once you’ve checked that the combined features work together, you might task your factory with building a phone-friendly interface so you can check invoices and record payments on the move.

The factory can also help run the invoicing workflow as well as improve its software. A scheduled agent could review overdue invoices, consult customer correspondence and prepare follow-ups for your approval. If that work repeatedly requires information the tracker doesn’t record, you can give the factory a task to add it. Building and operating feed into one another: agents use the software, and what happens in use informs its next improvement. This brings us to continual learning.

4. Continual learning

Your factory can also get better at how it works. Suppose payments disappear after restarting the invoice tracker, even though the agent reported that its checks passed. You’ll want it to fix the bug, of course. But why did its checks miss the problem?

Keep a record of assignments, checks and difficulties so a review agent can identify recurring problems and propose improvements. Did the worker lack a way to restart the app, or did it simply skip that check?

If it skipped the check, the agent could propose a skill that saves a payment, restarts the app and verifies the remaining balance whenever a change affects stored records. Test it against the original failure, then on further tasks. Does it catch problems with stored customer details too? Does it avoid unnecessary checks when a task only changes a heading?

The results help you decide whether to adopt, revise or reject the method. Apply changes within the authority you’ve agreed and keep them reversible. Warp’s self-improving factories illustrate this approach: failed checks can lead to proposed changes to prompts, skills or configuration, submitted for human review with evidence from the runs behind them.

Reassess methods as the project and model change. Anthropic’s work on long-running application development describes removing procedures as model capabilities improved and checking the effect.

A saved lesson needs to reach later workers and improve their results. This learning happens through changes to the instructions, tools and methods around the model, without requiring the model itself to be retrained.

What would you build?

Your factory could build a research tool that collects sources and lets you assemble briefings, a family intranet for shared plans and documents, or a private community space. Start with something useful to you, then give the factory further work as you discover what’s missing. Agents can also help operate what you build, from preparing a research briefing to organising incoming material.

Hugo uses this combination to run Vanishing Gradients, his media and education business. After a guest prep call, an automation retrieves the transcript and prepares interview topics, questions, titles and thumbnail options for review. After a livestream, agents can propose clips, cut video, add captions and prepare upload drafts. Saved editorial guidance and skills shape the work; Hugo chooses what to publish.

He also maintains the system, improving checks when agents miss instructions and updating skills based on results. More autonomous continual learning remains something he’s exploring. He describes the setup in How I Built a Media Empire With an Agentic Software Factory.

What’s one simple task you do repeatedly for your community, audience or business that you could start automating?

Decide what your factory needs next

Look at what keeps bringing you back to the work. If the agent cannot run the app or inspect a failure, improve the harness. If you keep repeating project context and decisions, personalise it. If development waits for you to start each task or manage every handover, add automation. If the same failures recur across runs, build a learning loop that changes how later work is done.

Start with work you care about and inspect what comes back. One agent can use all four layers. Add capabilities where they help you delegate more useful work, while keeping specification and verification part of the process throughout.

Build your own software factory

Ready to put these ideas to work? Hugo and Eleanor teach Build Your Agentic Software Factory, a two-week course where you’ll build a factory around something you want to create.

You’ll put the practices in this essay to use: give agents direction, set them working in the background, and learn to inspect and verify what comes back. You’ll leave with software of your own and a process you can keep using to build for yourself, your team or your business.

Bring an idea you’re excited about, or choose a provided starter project. No coding experience is required. The course runs October 13–22, 2026, with two live workshops, office hours for help with your setup and project, and a community of fellow builders.

Every student also gets $500 in Modal credits to run agent jobs in the cloud, experiment with open-weight models, or host something they’ve built. Your factory can keep working while your laptop is off.

Hugo and Eleanor are kindly offering my community 25% off, bringing the price from $950 to $712.50 USD. The link below applies the discount automatically, or you can enter FRIENDSOFPAUL at checkout.

Join Hugo and Eleanor and build your factory.

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

Images & videos

If not otherwise stated, all images and videos are created by the author.

“When I change it, what if something else breaks?” That’s what Alexey Grigorev told me about the long system prompt behind his RAG app. If you’ve ever hesitated before editing a prompt that works, you know the feeling.

Here is the thing… His first AI app version shipped and works fine. Then corrections pile up, and every fix grows the prompt into a huge monolith that rigidly handles each use case. Alexey wanted to clean it up, since it clearly adds a lot of noise, but without a way to measure the impact of trimming parts of it, he was too afraid to do it.

That’s why he had to add evals to his RAG app. Otherwise, you find out something broke only when a user tells you. As Alexey puts it: “without evals you are blind.”

But now we have another problem...

When you finally want evals, you discover you never recorded the interactions you’d need. Most people don’t add observability like Opik on day zero, so “build a gold dataset first” becomes a dream that never happens.

We will walk you through Alexey’s end-to-end eval workflow for his RAG app, built from proxy data when he had no traces. His RAG app is a FAQ chatbot integrated into his Slack DataTalks.Club community, which runs free courses (the Zoomcamps) for 100k+ learners. A perfect scenario for real-world RAG evals.

First, we will quickly look at the RAG app we will work with. Then we will focus entirely on the 5 steps you need to evaluate it: collect the data, build the eval set, build the evaluators, run them, and close the loop.

The RAG app: a FAQ system built on GitHub issues

DataTalks.Club courses are free, so thousands of people ask questions in their Slack. Some are unique, but “this is not the case for 99% of the questions.” At the top of the list is “I just discovered the course, can I still join?”

From an AI engineering perspective, their FAQ system is a RAG app that indexes all student questions into a light on-disk index that can be queried from Slack via the :faq: tag. A student asks a question; Alexey marks it with the tag; then a webhook triggers an AWS Lambda function that retrieves the top-k FAQs from the questions index, passes them to an LLM that compiles the answer, and posts the answer back in Slack.

The system is available in the DataTalksClub GitHub repository, along with the FAQ dataset, where we have one Markdown file per question.

During ingestion, a “FAQ proposal” issue triggers a GitHub Action that retrieves similar questions with minsearch, Alexey’s lightweight text-search library. It then asks an LLM with structured output for one decision: NEW, UPDATE, DUPLICATE, or WRONG_COURSE. NEW and UPDATE open a pull request Alexey reviews, while DUPLICATE and WRONG_COURSE close the issue. When looking for similar questions, it searches using both the question and the full record, merging all records via Reciprocal Rank Fusion (RRF).

We are not here to see how this RAG app works (if interested, full breakdown here), but to see how we can evaluate it, starting with the most important part: building the AI evals dataset.

Step 1: Collect the data, even if you never traced it

Alexey is “totally for vibe checks” early on. But from the very beginning, while you’re still vibe coding, ask your AI assistant to record every interaction, and your future eval set collects itself.

Alexey didn’t: “My goal was to just get this out and test how it works.” It worked. Then corrections piled up, but the system prompt got too big and noisy, and he was too afraid to touch it for fear of breaking something.

Today it’s stored in Opik’s prompt library:

🚨 His core problem was that he had no traces to build the evals dataset from!

So he used proxy data. Every FAQ contribution was a GitHub issue, and every correction was a commit on a PR. Issues give the inputs, and his corrections tell him what the output should have been. This meant he had access to the input, generated output, and expected output triplets he needed for evals.

How about reading the data yourself or being lazy (or optimal!) and using an LLM to do so? Alexey does “Both.” He lets PRs pile up for a few weeks, then clears them in one session with his AI coding assistant and a custom clear-backlog skill. The assistant proposes “merge,” “fix this first,” or “this case is interesting, let’s add it to our evals.” His job: “I just need to agree or disagree.”

Using the proxy data to fix his cold-start problem was only a temporary solution. Today, both the curation automation and the Slack assistant send traces to Opik: inputs, retrieved documents, outputs.

Here is how the Slack assistant traces look in Opik:

We have similar traces for the FAQ curation ingestion automation, but in a different Opik project. It works for both queries and ingestion.

For your RAG app, look for any place a human already judged your output (edited PRs, support tickets, corrected answers, thumbs-down). Then trace everything from now on, so you won’t need proxies again.

Step 2: Build the eval set

When building your dataset, the hardest part is labeling it with pass/fail scores, plus a critique. In Alexey’s scenario, every past decision told him what the right answer was, in 2 main ways:

• A PR he accepted unchanged is a “pass” sample.

• A PR where he had to alter it in any way becomes a “fail”, adding a one-sentence critique about why the review wasn’t accepted.

A false DUPLICATE (or WRONG_COURSE) is the dangerous error. The issue closes automatically, and no PR is opened, “so I would never discover it.”

The second biggest issue is replicating the context your RAG app used when ingesting/retrieving an item.

In our FAQ app, a question that was new when the issue arrived is now merged. Replay it as-is and the system correctly says DUPLICATE, so the eval case fails for the wrong reason. As I said during the interview, you have to “install your data,” not just your dependencies.

His answer is leave-one-out, shown in the image below. With 200 records within the dataset, where record D came from the issue under test, he removes D, replays the issue against the other 199, and expects NEW.

Each eval case is a few lines of Python code in Git: case ID, question, answer, expected action, and the ID of the doc to remove. The case ID is the GitHub issue number, so every case traces back to its issue or PR. See all here.

EvalCase(
    course="llm-zoomcamp",
    case_id=274,
    question="OpenRouter: Error code 402 when calling responses.create (max_output_tokens)",
    answer="""OpenRouter can return APIStatusError with code 402 when responses.create() 
is called without a max_output_tokens limit. Pass a lower limit: max_output_tokens=1024.""",
    expected_action="NEW",
    expected_section="module-1-homework",
    description="OpenRouter 402 error — valid NEW for module-1-homework",
    checks=[action_is("NEW"), section_is("module-1-homework")],
    tags=["correct-new"],
    relevant_doc_id="cfb07a27d5",
)

Every run costs money, so he trimmed the eval dataset to a small core set: 61 cases, with 38 NEW, 10 DUPLICATE, 7 WRONG_COURSE, 4 not WRONG_COURSE, and 2 UPDATE.

All the cases are imported into Opik as a versioned dataset for running the experiments:

For your RAG app, an eval set is inputs plus expected outcomes, weighted towards the errors that cost you most, and replayed against the knowledge base as it was when each input arrived.

Step 3: Build the evaluators

I asked Alexey, “How did you decide what metrics actually to measure?” His answer was per case type. Cases are grouped “based on the outcome I want”: correctly new, true duplicate, not a duplicate, wrong course. Each type “has its stack and then different logic for checking.” A “correct new” case checks both that the question is flagged as NEW by the LLM and that it is ingested in the right section.

His curation automation returns a structured decision, so the evaluators are deterministic: action match (right decision) and placement match (right section), plus the cost of each case. When he analyzed all his traces, the most common error turned out to be incorrect section placement. Instead of placing a record about projects in the “Project” section, it would place it in “General”.

One curation trace flagged as DUPLICATE with its structured decision, model, and cost:

These evaluators need no LLM, are cheap to run, and are never ambiguous. Retrieval is checked separately: for a given query, it verifies that it retrieves the expected chunks. A 25-case suite scored by recall@5 runs in about 2 seconds, while the 61-case generation suite takes about 2 minutes.

The Slack assistant’s answers have no single right string. There, you need an LLM judge aligned with your own labels. Label 10 to 15 examples as good or bad yourself, write the rules down, and iterate until the judge agrees with you, as Alexey lays out in his evals guide.

For your RAG app, use code checks wherever the right answer is known (a decision, a label, a retrieved document), check retrieval separately from generation, and save LLM judges for free-text answers.

Next, I asked him how he decides whether an eval experiment passed.

Step 4: Run the evaluators in your eval harness

First, let’s see how the eval harness that runs the experiments works.

Alexey keeps the eval cases in Git as the source of truth, and uses Opik as the eval harness: he uploads that dataset version and attaches it to each experiment, so every result shows what it ran on.

His runner is plain code in Git that runs all cases, or just one, and logs each run to Opik as an experiment with per-case action match, placement match, and cost. He admits it’s “not very scientific” yet and wants to track runs historically in Opik.

Here is one eval run, as an Opik experiment, where we can see the aggregated and per-item scores:

“I never needed 100 percent,” Alexey says, because his AI assistant still reviews every PR. The number is a benchmark that must not go down. Today the generation eval set passes 42 of 61 cases on GPT-5.4 nano. When new cases make it dip, he changes nothing else and takes it as the new baseline. Hard cases can sit in FAIL: he doesn’t want the system prompt to grow to handle every corner case, but he does want to know those cases exist.

He pays for this community project out of his own pocket and wants “the cheapest possible way that is performant enough to be useful.”

With evals, he compared models: GPT-4o mini is “very cheap but stupid” for this decision, so he chose GPT-5.4 nano, the cheapest that still delivers good quality. He keeps $5 in his OpenAI account and forgets about it “for half a year or even more,” because “each query costs on average less than one cent,” as the metrics from Opik show.

When the automation misbehaves, he first captures the failure as an eval case. He runs the new prompt on that case until it passes, then on the whole set to check nothing else broke. “It’s kind of test-driven development, but like eval-driven development.”

By tracking each run as an Opik experiment, we can clearly see the difference in scores in the before, after, and verify runs tracked over time:

For your RAG app, keep the cases in version control, run each experiment against a versioned dataset in your eval harness, always keep one baseline number as reference, and turn every failure into a case before you fix it.

Step 5: Close the loop in Opik

Mining Git history worked once, but “it’s not the exercise I want to perform every time.” Now, when he sees a bad Slack reply or someone corrects the bot, that question becomes an eval case. He uses his coding agent to pull the trace from Opik into his eval set.

Below is Alexey’s Slack assistant correction dataset:

“Because I have evals … I can see where it’s failing,” and he usually fixes it by improving the FAQ, the docs, or the course repository, not with a fancier model. Then he re-runs the evals.

Alexey calls evals “basic hygiene” that you know is good for you, but you still skip it from time to time. Everyone knows you should go to bed at the same time, and sometimes you still go at 2 a.m. instead of 10 p.m.

For your RAG app, route every user correction straight from your traces into the eval set instead of reconstructing it from history.

In my experience, integrating Opik takes about 5 minutes, and the free tier covers 25k spans a month, which, for this project, Alexey “can use free forever” (a serious production app might burn that in a day). He tried the open-source version first, but with his lean infra, self-hosting would have been more expensive.

Opik didn’t fit in 2 places. His evals remove records from the dataset per case, and he found no straightforward way to run that inside Opik, though “maybe it’s just I should spend more time on understanding.” And his zero-dependency Lambda couldn’t easily take Opik’s default Python client, so his AI assistant switched to plain HTTP requests, which “was so simple at the end.”

Final Thoughts

“I have so many applications with zero evals and a bunch of vibe-coded test cases I don’t even know test the right thing. Is that wrong?”
— Paul

Alexey’s answer: there’s nothing wrong with it, as long as you know how to build an eval set when you need one. First figure out how the app should work and what the UX is. If you pour time into evals and then learn the UX is wrong, you throw the evals away with it.

“Many engineers who don’t know the domain look blindly at the logs and can’t decide: is this good, is this bad, what am I looking at?”
— Paul

His advice: sit down with the domain experts. Even when he was a data scientist, long before AI automation, the most useful thing he did in every job was learn from the people his models were built for, because their input decides what “good” means.

That’s why I think humans still sit in evals. A coding agent can build the app, but it won’t tell you what it costs to run, how slow it is, or whether it’s actually good. Start with Alexey’s How to Do Evals in 2026, then Eugene Yan on aligning LLM evaluators and Hamel Husain on why AI products need evals.

But here is what I’m wondering:

How are you evaluating your RAG workflow or agent?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

Thanks again to Opik for sponsoring this case study and keeping it free!

If you want to monitor, evaluate, and optimize your AI workflows and agents:

Try Opik for free

Images

If not otherwise stated, all images are created by the author.

Every AI application that wraps an agent is a harness!

In LangChain’s Terminal-Bench experiment, changing only the harness (with the same model) moved a coding agent from ~30th place into the top 5: the harness, not the model, is what makes a coding agent good.

In the open-source course Building a Coding Agent From Scratch, you’ll build that harness from scratch in Python: Decode, a complete coding agent that grows lesson by lesson from a bare agent loop into a swarm of remote agents running in parallel in the cloud.

Why? You’ll be able to engineer custom harnesses for your own AI products (the skill behind that leaderboard jump), and you’ll understand what Claude Code and Codex actually do under the hood, turning you into a power user.

Lessons:

1. Building a Coding Agent From Scratch

2. The Bare-Bones Coding Agent Loop

3. From a Raw Shell to a Sandboxed Coding Agent

4. Context Engineering for Coding Agents

5. Subagents Are Context Engineering

6. Swarm of Remote Agents

7. Agent Evals 101

8. Turn Your Agent’s Failures Into a Regression Suite ← you are here

Full open-source course

Special thanks to Modal, Opik (by Comet), and Kitaru (by ZenML) for sponsoring this open-source course and keeping it free!

Lesson 8: Turn Your Agent’s Failures Into a Regression Suite

Here I am, at 10 p.m., creating my implementation plan for my software factory. Tired, I vibe-planned more than usual, using Claude Code to implement Decode’s evals layer via my Modal serverless endpoint so I could access all the necessary LLMs. The endpoint returned a 503 before it finished warming up, and instead of stopping, the coding agent found a Gemini API key in memory and decided to use it to achieve its goal no matter what. The result: $40 of tokens burned on 20 benchmark tests by the next morning, compared to $4 to $5 on Modal. Ouch.

The fix was extremely simple, just 2 lines: wait for the warm-up and properly guard my credentials from the env vars. The hard part is guaranteeing that the same error never happens again after the next prompt, tool, or model change.

Without that guarantee, you’re afraid to change your prompts, because the chance of breaking what worked last week without noticing is too high. That slows development down, especially in larger teams where you have to touch other people’s code.

For an agent, the best way to organically grow your regression tests without spending too much time guessing them or money on testing useless errors is the error analysis framework.

To apply the error analysis framework to your production or synthetic traces, you need to build an eval harness that runs your agents in parallel in isolated containers, emulates the right context for each test, and collects the outputs from all your tests.

For that, there is a new family of tools in town, such as Kitaru by ZenML Labs, focused on capturing your agent runs and replaying them with the same context so you can easily reproduce bugs, fix your code, and grow and manage your regression tests.

First, you’ll get a strong intuition for how the error analysis framework works, then you’ll apply it to fix and build regression tests on top of Decode, the Claude Code clone we’ve built during this series. This article will make sense even if you haven’t read previous lessons. Still, reading Lesson 7 on Agent Evals 101 would definitely help you get the foundations of evals.

Regression tests via the error analysis framework

Every change to a coding agent mixes plain English (prompts, tool descriptions, skills) and code: tool implementations, harness components, the sandbox layer (the container its bash and file tools run in), and compaction. So implementing unit/integration tests as we know them won’t work for three reasons: it’s impossible to write unit tests for prompts, writing integration tests for too many scenarios is too costly, and you can’t assert unstructured data via code.

What works is the error analysis framework, popularized by Hamel Husain and Andrew Ng.

Here is how error analysis works, step-by-step.

You record real runs, read a sample by hand, and give each a pass/fail label with a short critique, never a 1-to-5 score, because a “fail” (instead of a “pass”) is actionable, while a “3” (instead of a “2” or “4”) is not. You fix the obvious right away, cluster the rest by failure type, rank by frequency × severity, and run a root cause analysis (tracing one failing run back to the step that broke it) on each high-priority cluster. Then you write one cheap evaluator per cluster and re-run it after every change.

Hamel Husain calls it “the most important activity in evals” because it decides which evals to write at all, and Anthropic’s field guide sets the entry bar low: “20-50 simple tasks drawn from real failures is a great start”.

Error analysis sounds powerful, but how do we do it in practice? How do we look at our data, cluster it and run evaluators?

That’s what we will show you in this lesson, using Kitaru.

When evaluating Decode, you record isolated decode run executions (Decode’s headless command) as sessions, with every LLM and tool call included. One Kitaru session maps to one Decode session, similar to threads or a collection of traces from other tools. Or, in plain English, your end-to-end conversation with an agent is a session. More here.

Ideally, you already have Kitaru plugged in while running your AI app in production. Otherwise, you can easily import all your data from observability tools like Opik, Braintrust, or Langfuse.

Next, you sample ~20-30 sessions into an investigation, label each session as pass/fail, and add a short critique message. For errors, the message should contain the first failure reason.

After that, because Kitaru connects to your coding agent through its MCP server and skills, the agent knows how to cluster the problematic ones by error type into cohorts. For each cohort, you will implement one evaluator, a few lines of Python that read a session and say pass or fail. You fix the code, then start an experiment that replays every session of the cohort with the fix in place.

The power of Kitaru is in its replay mechanism. Why? When implementing the eval harness, one of the hardest parts is recreating the same context for each task you want to evaluate. Via replays, you inject that context automatically by caching the tool outputs of the session you want to replay. This means that if a tool gets the same input, it returns the same output. Tool calls with different arguments, or tools changed during your fix, run from scratch.

Now let’s plug Kitaru into Decode’s harness to record, review and replay its agent loop.

Integrating Kitaru into the agent harness

Sessions can get into Kitaru in three different ways:

1. Kitaru is already plugged into your harness

2. You import sessions from an observability tool like Opik

3. You run experiments via Kitaru after fixing a bug or implementing a feature

In all three methods, everything is ingested into Kitaru’s control plane. Your coding agent talks to that control plane to do the error analysis, and experiments replay sessions on workers that you host yourself.

From an architectural perspective, 4 components talk to each other:

• Decode is the agent under test, available in the series repo.

• The Kitaru control plane stores sessions and tracks the error analysis: local open-source via Docker Compose, managed, or self-hosted.

• The Kitaru workers run Decode in isolated containers: via Docker when running locally, as Modal serverless apps when deployed, or any other method you prefer. The idea is that everything runs on your infra, and Kitaru doesn’t manage it.

• Your coding agent (Claude Code, Pi, Codex) does the error analysis through the Kitaru MCP server (the repo’s .mcp.json), guided by 3 Kitaru skills: kitaru-investigation, kitaru-validate-evaluator, kitaru-replay-experiment.

A worker, when executing an experiment or a replay, pulls each job on your infra and spawns one isolated Decode per session.

In the code snippet below, we can see how Decode plugs into our Pydantic AI agent using the wrap_for_recording function, which wraps the agent using KitaruAgent only when recording is configured, with no Kitaru import on the bare path.

It probes the workspace once, so an unreachable Kitaru server degrades to an unrecorded run with one warning line. The one exception the block leaves out is a replay spawned by a Kitaru worker, where the same failure is fatal because an unrecorded replay would waste tokens.

From src/decode/runtime/recording.py:

async def wrap_for_recording[DepsT, OutputT](
    agent: AbstractAgent[DepsT, OutputT], *, session_name: str | None = None
) -> tuple[AbstractAgent[DepsT, OutputT], str | None]:
    if not recording_is_configured():          
    # KITARU_AGENT_ID + KITARU_API_URL, else the bare agent
        return agent, None

    export_kitaru_api_url()
    try:
        agent_id = _configured_agent_id()
        ...
        await _probe_workspace(agent_id)       
        # Ping the workspace ONCE, before the run burns tokens
        from kitaru_pydantic_ai import KitaruAgent

        wrapped = KitaruAgent(agent, agent_id=agent_id, session_name=session_name)
    except Exception as error:  
    # Broad on purpose: ANY setup failure degrades to the bare agent
        ...
        notice = (
            f"[kitaru] not recording this run: {_workspace_label()} is unavailable "
            f"({one_line(error)}); continuing on the bare agent"
        )
        logger.warning("%s", notice)
        return agent, notice

    ...
    return wrapped, None

And this is how we call it, from src/decode/runtime/headless.py:

agent, recording_notice = await wrap_for_recording(
    _build_headless_agent(model), session_name=session_id
)

session_name is Decode’s own session id, so a Kitaru session and its Opik trace thread share one id, which is what makes the Opik import in the next section a one-to-one mapping.

With everything in place, let’s run the framework end to end.

From 30 runs to a regression suite

You need a mix of at least 20 to 30 sessions to see the agent loop in action. Either run decode run "<goal>" yourself, or let make kitaru-seed record 30 runs in ~10 minutes. You get 14 good sessions and 2 intentionally introduced error clusters: 8 cut off by --max-requests 1, and 8 crashed on a bogus model or a broken provider. The seed emulates 2 common errors we hit while developing Decode and stores all the threads in Opik.

You can find all the commands on how to replicate our experiment in our GitHub runbook.

You sample 20 of the newest 30 sessions into an investigation. For each one, you ask, “What do you notice? Did it match what should have happened?” and label it in the UI as acceptable, problematic, or uncertain, plus an optional critique.

I had 6 acceptable, 14 problematic, and 0 uncertain, which the investigations tab sums up in one row:

After wrapping up the investigation, the UI hands you a prompt to paste into a coding agent with the Kitaru skills and MCP server hooked up. I used Opus 5.5:

Investigation complete: my-discovery-1. 20 of 20 sessions reviewed, agent: decode.
Reviewed 23 September 2026.

Verdicts: 6 acceptable, 14 problematic, 0 uncertain.

Choose what to do next:
1. Build a cohort from a behavior found in these reviewed sessions.
2. Investigate the 14 problematic sessions in more detail.

Running the prompt finds 2 cohort candidates by error type: a run that hits the request limit and ends on a raw UsageLimitExceeded (the error Decode raises when a run hits its request cap) with no output, and a connection error, which it suggests rejecting as agent behavior. You answer “Create a cohort for each candidate!” and get 2 cohorts.

You either apply an existing evaluator or create one per cohort. We have one of each: decode_connection_error.py for the connection-error cohort and decode_request_limit.py for the request-limit cohort. A new failure class needs a new evaluator, which is easy because the cohort holds every session’s logs and errors, providing enough context to your coding agent.

💡 We always prioritize code checks over LLM judges. They are cheap, fast, and need no maintenance, while a judge needs labeled examples and calibration first. A judge is always the last resort.

Both evaluators read the session’s terminal error and flag the marker they own, and a non-terminal session with no error returns passed=None, never a silent fail. The evaluator fails when the failure class is present, so a cohort session fails until the code is properly fixed or the bug is accidentally reintroduced.

Anthropic warns that checking an exact tool-call sequence is “too rigid”, so check the overall contract, such as the outcome or a hard constraint like “the run must not end on this error”.

From evaluators/decode_connection_error.py:

from typing import Any

from kitaru.task.evaluator import EvaluationResult, SessionView

_RESULT_NAME = "connection_error_crash"
_MARKERS = ("Connection error", "ConnectError")


def evaluate(session: SessionView, **params: Any) -> EvaluationResult | list[EvaluationResult]:
    record = session.session
    error = record.error or ""

    if not error and str(record.status or "") == "in_progress":
        return EvaluationResult(
            name=_RESULT_NAME,
            passed=None,
            value="unresolved",
            explanation="Session is non-terminal with no error; evidence missing.",
        )

    detected = any(marker in error for marker in _MARKERS)
    explanation = (
        "Terminal error is a connection failure: the provider endpoint was unreachable."
        if detected
        else "No connection failure: the session has no error or failed for another reason."
    )
    return EvaluationResult(
        name=_RESULT_NAME, score=detected, passed=not detected, explanation=explanation
    )

The decode_request_limit evaluator, from evaluators/decode_request_limit.py, has the same Python logic, but with different markers:

_RESULT_NAME = "request_limit_cutoff"
_MARKERS = ("UsageLimitExceeded", "exceed the request_limit")

You prompt the coding agent to apply the 2 evaluators, one per cohort. Each session fails because it still contains the error.

Now you run a root cause analysis on one cohort session and fix the code behind it. Ours was artificially injected by the seed, so we know the rerun will pass. In a real scenario, you fix the code first.

This cohort is the failure class of my 503 error due to the missing warm-up logic when running my models on Modal, now frozen as 4 replayable sessions. After fixing the bug, you prompt, in the same session: “For cohort decode-connection-error we fixed the connection error. Start an experiment based on the sessions from the cohort, rerun the evaluator and see if the failure class has been resolved.”

Kitaru replays every cohort session on a worker in an isolated Decode instance, from the original prompt. These baselines crashed before their first tool call, so the experiment’s tool policy (next section) lets tools run for real.

Two runs: run 1 replays the cohort with the code untouched and still fails on every session, so the failure reproduces and the recording is trustworthy. Run 2, after the fix, shows the failure class resolved on every session.

Properly understanding replays

A replay re-runs Decode from scratch. The only exception is that all tool calls are recorded, which means that if an agent calls a tool with the same inputs, it will use a cached output.

This means the agent loop’s decision path reflects the latest code changes. To avoid recreating the same context used by the tools, we reuse cached tool outputs instead, making it 10x easier to run experiments in the same context than to reproduce the files or database state the agent had at that moment. In Kitaru’s terms, a replay is still a fresh execution, not a playback.

Replays are an amazing way to reproduce your bugs as well. Start from your observed session, reproduce it with 0 changes to the code, then create a fork, change your model, system prompt or other parts of the code and run a replay again. Then you can do a diff between the reproduction and the fork to see the impact of your change.

Remember that every replay, as it executes your agent, burns real tokens instead of mocking the calls, so a 4-session cohort is cheap to rerun on every change and a 200-session one is not. That’s why cohorts stay small, and ideally evaluators are code, not judges.

In reality, we have more control than always using the cached outputs. For each tool (bash, file writes, web fetches), the tool policy decides whether the call is answered from the recording, a fixed table, a real execution, or a model, and what happens on a miss. Without a suitable policy, “a replay can call a live tool and repeat its side effects”: with the default passthrough, a replayed refund_payment refunds the card again.

Tool policy configs:

• type: history (answer from the recording), static (a fixed table), passthrough (run the tool for real), or llm (let a model invent the result).

• on_miss: what happens when the recording has no answer: fail (stop the replay), error_result (hand the model an error and continue), or passthrough (run the tool for real).

• scope: which recordings history may answer from: baseline, cohort_version, or agent.

Pick on_miss (what a replay does when the recording has no answer for a tool call) by how complete the recording is. Complete baselines get history with on_miss: fail, so an unanswerable call stops the replay instead of inventing the rest. Cut-off or crashed baselines, like our 2 cohorts, need on_miss: passthrough.

The tool then runs live inside the worker’s workspace (the fresh clone a worker runs Decode in). That is safe for Decode only because its live tools are bash and file edits inside a throwaway container. For an agent whose tools touch real resources such as a payment card or a database, passthrough is extremely dangerous and needs to be carefully mocked.

A replay always starts from a single session and produces a new one, listed next to the original in the sessions tab with Replay as its origin:

Select both, and the dashboard diffs the replay against its baseline, node by node:

A replay with one change is exactly how you test a change to the harness. Let’s see how that works when swapping the model.

Swap the model, replay the same cohort

Suppose you change the harness and want to see how that change affects performance. Here the change is the model: from Qwen/Qwen3.6-35B-A3B-FP8 on Modal to gemini-3.8-flash.

You create the experiment (nothing runs yet) with the following config: the decode-connection-error evaluator, a tool policy of history on the baseline with on_miss: passthrough, and a model override that replaces the model on every model call. The same experiment shape tests a prompt change through the system_prompt or prompt keys, or a parameter change through model_params.

You start the run against the cohort version you replayed before (the frozen list of session ids the experiment ran against), and the worker replays each session on the new model.

When it finishes, you read the pass/fail counts for that run. All 4 sessions passed:

And the full experiment config sits next to them, with the model overridden and every tool answered from the recording:

⚠️ decode-connection-error only flags connection errors, so a replay that crashes for any other reason still passes it. A green run shows the new model did not reintroduce connection failures, not that the migration works. Pair it with an evaluator that checks the task itself, and read pass/fail per session plus the cost delta, never one aggregate score.

What replays do not solve

Replays are a good strategy for reproducing behavior and running regression tests, but not for running benchmarks that measure performance and optimize your system. In that case, run everything from scratch in isolated environments on open-ended tasks that ask whether the agent can complete its goal at all. See Lesson 7 to learn more about implementing your custom benchmark.

Replays also do not discover unseen scenarios. A regression suite only catches previous errors detected via the error analysis framework.

🧑‍💻 Clone the course repo, follow the replays runbook, seed 30 sessions, label 20, pick a cohort and build an evaluator for it.

With this lesson, we’ve finally wrapped up the How to Build a Coding Agent From Scratch course, where we’ve learned how to design, build and evaluate a Claude Code-style coding agent.

For previous lessons, here is the lesson-by-lesson course roadmap (see all on GitHub):

1. Building a Coding Agent From Scratch

2. The Bare-Bones Coding Agent Loop

3. From a Raw Shell to a Sandboxed Coding Agent

4. Context Engineering for Coding Agents

5. Subagents Are Context Engineering

6. Swarm of Remote Agents

7. Agent Evals 101

8. Turn Your Agent’s Failures Into a Regression Suite ← you are here

But here is what I’m wondering:

What is your current strategy to grow your regression tests?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

Special thanks to Modal, Opik (by Comet), and Kitaru (by ZenML) for sponsoring this open-source course and keeping it free!

Images & videos

If not otherwise stated, all images and videos are created by the author.

Every AI application that wraps an agent is a harness!

In LangChain’s Terminal-Bench experiment, changing only the harness (with the same model) moved a coding agent from ~30th place into the top 5: the harness, not the model, is what makes a coding agent good.

In the open-source course Building a Coding Agent From Scratch, you’ll build that harness from scratch in Python: Decode, a complete coding agent that grows lesson by lesson from a bare agent loop into a swarm of remote agents running in parallel in the cloud.

Why? You’ll be able to engineer custom harnesses for your own AI products (the skill behind that leaderboard jump), and you’ll understand what Claude Code and Codex actually do under the hood, turning you into a power user.

Lessons:

1. Building a Coding Agent From Scratch

2. The Bare-Bones Coding Agent Loop

3. From a Raw Shell to a Sandboxed Coding Agent

4. Context Engineering for Coding Agents

5. Subagents Are Context Engineering

6. Swarm of Remote Agents

7. Agent Evals 101 ← you are here

8. Turn Your Agent’s Failures Into a Regression Suite

Full open-source course

Special thanks to Modal, Opik (by Comet), and Kitaru (by ZenML) for sponsoring this open-source course and keeping it free!

Lesson 7: Agent Evals 101

While optimizing my coding agent on my custom benchmark, I compared its performance against two models: a 35B vs. a 120B. The winner should be obvious, right? As the 120B model is 3 to 4 times larger (both are MoE). Well... the 35B model had a 95% success rate, while the 120B model had 53%. Auch.

There is more to this story, which I will explain in this article, but this is a clear signal that public benchmarks, model size, or other model features mean nothing until you test them on your custom evals built for your use case.

In this article, I want to show you exactly that. How to build a custom eval harness for your agent harness (word overloading alert!) to quantify your application’s performance, cost and latency on what you care about. Delivering shareholder value... Or if you are like me, just to create a badass AI app that you actually know works with concrete metrics, not just assumptions and vibes.

To avoid staying too abstract, I will show you how to build the evals layer for Decode, my Claude Code educational clone I built for this series. Still, this piece makes sense on its own, treating the coding agent as a black box. Plus, we will focus on techniques that can easily be extrapolated to other types of harnesses such as research agents or personal assistants. Ultimately, the core of the harness is the same regardless of the domain it attacks.

In reality, evals are more than benchmarks. That’s why it can get confusing, since it has so many facets. So let’s start by understanding the 3 big ways to think about evals and how to integrate them into your AI app. Then, we will zoom in on designing and building them.

Three questions every production agent must answer

We run Decode as a background agent by typing decode run in the terminal. The eval harness uses the same interface to run it as a subprocess, so we can leverage the sandbox (the container its bash and file tools run in) and hand-back mechanism (the agent returns its work as a git branch) already baked into the coding agent to isolate the agent while being executed within the eval harness.

From evals/harness/trial.py:

command = [
    sys.executable, "-m", "decode", "run",
    task.instruction,                    # inject trial instruction
    "--repo", str(seed_dir),             # where to get the code from
    "--local",
    "--max-requests", str(task.max_steps),
    "--summary-json", str(agent_dir / "summary.json"),
]
process = subprocess.Popen(
    command,
    cwd=home_dir,                        # working directory of the trial
    env=child_env(sandbox=sandbox),      # SANDBOX_MODE=docker|modal,
    start_new_session=True
)

The --repo flag clones the seed repo (a fresh git repo at a pinned commit) so every trial starts identical, and --summary-json outputs the run’s report, so grading never parses traces, but loads a structured report that can be interpreted in Python or other programming language.

The eval harness never imports the agent’s loop. It launches the agent in a different process as a completely different program through the narrowest interface the product has (a command line, an HTTP call, a function). So the eval and coding harnesses are two different programs, completely decoupled.

Benchmarks answer “does it work?” They run during feature development. These are the metrics you optimize every feature against. It helps you understand if a prompt tweak truly helps or not.

Regression tests answer “do previous features still work?” after making changes to your codebase. When you change system prompts to optimize bash tool calls, they help measure how that affects the agent’s recall on other tools.

Online evals answer “does it work as expected?” on live traffic samples from production.

In a coding agent, swapping Kimi K3 for GLM 5.2 can improve our benchmark scores but fail the regression that checks whether the read tool is triggered when skills point to subfiles that should be read via progressive disclosure.

Or in a research agent, a prompt tweak that makes it stop citing sources passes every benchmark it was optimized on, but fails the regression case that asserts “every claim cites a document”.

I have an article on integrating AI evals into an AI app for the 3 stages in depth.

Now, let’s take a look at the system design of the eval harness.

The system design of an evaluation harness

Any benchmark trial from Decode runs through 5 phases: seed → run → collect → verify → record. Collect reads summary.json as the report from the git branch the agent handed back. Record writes result.json to the trial dir, where we collect all the results.

Within an agent harness, “Harness” usually means everything around the model: harness = agent - model. An eval harness is a separate application on top of that agent. Its sole purpose is to execute and evaluate the agent harness within the right environment and context.

Our educational harness is inspired by Harbor, one of the most popular eval frameworks used to compute the Terminal-Bench (general agentic computer use) and DeepSWE (agentic coding) benchmarks.

They frame the eval harness as the referee and the agent under test as the player. Harbor’s rule: “the agent never grades itself and never provisions itself”. In Decode’s implementation, this can easily be achieved by always calling the agent as a subprocess via decode run as seen in the previous section.

We used the same vocabulary as in Harbor. A task is an instruction, an environment, and a hidden oracle. A dataset is a collection of tasks. A trial is a rollout of a Task that produces a reward. A job is many trials run in parallel.

The idea is to keep this design flexible enough to use either Harbor or a custom version. In our implementation, we used Opik (open-source) as our evals platform. Beyond capturing and storing all the agent’s traces, it lets us store and version datasets, run jobs, and wrap the whole run as an experiment. Extremely useful for lineage and comparing multiple runs.

Below, we see how we defined a job run via Opik where we define the "decode-benchmark" dataset, the task function that executes the coding agent for each item in the dataset, the metrics, and other job details. The scoring metric is the 0 or 1 reward emitted at the trial level, while the experiment scoring function is used at the experiment level to compute aggregate metrics such as pass@k (at least one of the k trials per task passed), pass^k (all k trials pass) and flakiness (pass@k − pass^k, the tasks that pass sometimes but not always, which the agent can solve but not reliably).

From evals/harness/benchmark.py:

from opik.evaluation import evaluate


task_fn = make_benchmark_task_fn(
        {task.id: task for task in all_tasks},
        sandbox=sandbox,
        job_dir=job_dir,
        model=model
    )

result = evaluate(
    dataset="decode-benchmark",
    task=task_fn,                                      # one trial
    scoring_metrics=[RewardMetric()],                  # the verifier's reward function that outputs 0 or 1
    experiment_scoring_functions=EXPERIMENT_SCORING_FUNCTIONS,   # pass@k, pass^k, flakiness,
    experiment_config=experiment_config(
        sandbox=sandbox, trials=trials, model=model
    ),
    experiment_name=job_name,
    dataset_item_ids=list(item_ids.values()),          # only the matching items
    task_threads=task_threads,
    trial_count=trials,                                # --trials k → pass@k
)

Example with k=3 and 10 tasks: 7 tasks pass all 3 trials, 2 pass 1 or 2 trials, and 1 never passes. That gives pass@3 = 90%, pass^3 = 70% and flakiness = 20%.

When k=1, all three collapse to pass@1, and flakiness is 0, as the Opik experiment below shows:

On top of that, it is good to track error types, so you never mistake an infrastructure issue for a reward of 0:

• infra_error: the harness failed (seed, clone, a crashed verifier).

• agent_fail: the agent failed (wrong answer, request ceiling, timeout).

Within Opik, we keep all our data: traces, datasets, experiments, test suites, and online rules. The actual execution of the benchmark is done on our end via our custom eval harness (or Harbor).

Other popular eval platforms include LangSmith, Langfuse, Braintrust, and Azure AI Foundry. If you have the right resources (and knowledge!), you can also consider building your own infrastructure. Especially now, it’s quite easy to one-shot a simple observability tool with a coding agent as long as you have a hosted database in place. But still, starting with one of these tools is the easiest path. Especially because tools like Opik are open-source. The only reason to go custom is if they are too rigid for your use case, a common scenario being your data format.

Now, let’s see how we designed our benchmark and regression tests.

Designing a benchmark that predicts your agent

The best way to understand how the benchmark works is by looking at one example: 007-fix-failing-test: the agent gets an instruction and a seed repository, hands back a git branch, and the verifier runs the one fail-to-pass test, which returns a reward of 0 or 1.

Here is how the task is structured, where the agent gets a buggy environment it needs to fix:

evals/benchmark/tasks/007-fix-failing-test/
├── task.toml            # step budget, timeouts, fail_to_pass / pass_to_pass
├── instruction.md       # the agent's prompt
├── environment/
│   ├── ranges.py        # the buggy module
│   └── test_ranges.py   # the test suite
├── tests/
│   ├── test.sh          # the Verifier: writes $VERIFIER_DIR/reward.txt
│   └── test_ranges.py   # the HIDDEN test suite
└── solution/            # the oracle solution
    ├── solve.sh          
    └── ranges.py

The domain-agnostic contract: an instruction, a frozen environment, and a hidden verifier program that turns the end state into one reward. We use the same layout as Terminal-Benchwhere the verifier runs in isolation after the agent finishes.

Each trial is its own decode run subprocess, with the agent’s workspace isolated in a Docker container (or a Modal sandbox) via the sandboxing logic from lesson 3. When the agent finishes, its work is handed back as a git branch. The host grades a clean clone of that branch with the hidden tests copied in last (to avoid leaking the solution), then aggregates the rewards across trials.

For a research agent, we could define a task as a question, a seeded corpus, and a hidden code verifier that checks whether the agent used the right citations against a ground-truth file, instead of a costly and flaky LLM judge.

tests/ is the verifier (the hidden program that turns a trial into one reward), injected only after the run. solution/ is the oracle, a reference that, if everything works well, outputs a reward of 1. We use it within our CI pipeline to test our eval harness, alongside running it on the unfixed environment, which should have a reward of 9.

Here is what the task.toml file from the 007-fix-failing-test task looks like:

From evals/benchmark/tasks/007-fix-failing-test/task.toml:

[task]
name = "007-fix-failing-test"
description = "Fix the bug in a small module so its whole unittest suite passes."

[metadata]
difficulty = "easy"          # easy 15 steps / 600 s · medium 25 / 900 s · hard 40 / 1500 s
category = "Software"        # Terminal-Bench's taxonomy

[agent]
timeout_sec = 600.0
max_steps = 15               # decode run --max-requests

[verifier.tests]             # SWE-bench shape: the test the fix must flip, the tests it must not break
fail_to_pass = ["test_ranges.TestRanges.test_inclusive_end"]
pass_to_pass = ["test_ranges.TestRanges.test_starts_at_one", "test_ranges.TestRanges.test_contains_two", "test_ranges.TestRanges.test_no_zero"]

And here is the test.sh file that emits the reward. Custom for each task, where every exit path must still write exactly one reward.

From evals/benchmark/tasks/007-fix-failing-test/tests/test.sh:

(
  cd tests && python3 -m unittest -v \
    test_ranges.TestRanges.test_inclusive_end \
    test_ranges.TestRanges.test_starts_at_one \
    test_ranges.TestRanges.test_contains_two \
    test_ranges.TestRanges.test_no_zero
)
status=$?
reward=0
[[ $status -eq 0 ]] && reward=1
printf '%s\n' "$reward" > "$VERIFIER_DIR/reward.txt"

In our benchmark, we have a total of 19 tasks, while Terminal-Bench has 66 (see them here) and DeepSWE has 117 (see them here). In this case, more is NOT better! The idea is not to have hundreds of tests, since they can get expensive to run, but to have tests as diverse as possible, capturing all your scenarios and edge cases. This is known as your core set. So you need to think through each task carefully, designing each environment, prompt, and test file that measures the outcome.

In other words, creating a high-signal benchmark that’s actually useful is not that easy. That’s why regression tests, as we will soon see, are an easy win relative to benchmarks.

In our Decode benchmark, we have 7 easy tasks, 6 medium, and 6 hard.

We stored and versioned the whole dataset in Opik, which makes it easy to share across runs or people:

Task 015-secret-scrub asks the agent to move two hardcoded secrets into environment variables, and the first check of its verifier diffs the handed-back branch against the seed repo’s base commit. It fails the trial if any file other than service.py changed or if service.py changed by more than 8 added-plus-deleted lines, which turns “keep the diff minimal” from a judge’s opinion into a measured bound.

From evals/benchmark/tasks/015-secret-scrub/tests/test.sh:

MAX_CHANGED_LINES, CHANGED_FILE = 8, "service.py"
base = git("rev-list", "--max-parents=0", "HEAD").split()[0]        # the seed Repo's base commit
changed: dict[str, int] = {}
for line in git("diff", "--numstat", base).splitlines():            # added, deleted, path — per file
    added, deleted, path = line.split("\t", 2)
    if not ignored(path):                                           # tests/, .verifier/, .git/ don't count
        changed[path] = changed.get(path, 0) + int(added) + int(deleted)
extra = sorted(p for p in changed if p != CHANGED_FILE)
if extra:
    print(f"FAIL: only {CHANGED_FILE} may change, but the answer also touched {extra}"); sys.exit(1)
if changed.get(CHANGED_FILE, 0) > MAX_CHANGED_LINES:
    print(f"FAIL: the change to {CHANGED_FILE} is more than the {MAX_CHANGED_LINES} lines a minimal fix needs"); sys.exit(1)

The beautiful part of putting all this effort into a benchmark is that once it exists, it gives the coding agent so much signal that we can start optimizing our agent on automode. We just give the coding agent the goal of tweaking the harness until it passes all the trials, similar to TDD. Who knows, maybe in the future, all we will do is design benchmarks and let coding agents do the rest.

When using Opik, you can use their Agent Optimizer to vary prompts, tool descriptions, and model parameters; score each trial against the dataset into an experiment; compare them; pick the best one; and repeat. Or run a similar loop by plugging Opik’s experiments into your coding agent via their MCP server.

Here is how you can run the benchmark job over Decode using Modal to host a Qwen3.6-35B model, where all the results will be saved as an Opik experiment:

LLM_PROVIDER=modal make eval-benchmark ARGS='--threads 1 --job-name qwen36-modal'

It scores 95% pass@1 across all 19 tasks:

And this is what it looks like when we compare two experiments to test how the harness behaves with a larger model (GPT-OSS-120B) against the default model we optimized the harness for (Qwen3.6-35B):

The table below, run via a Modal endpoint with 1 trial per task, shows that the larger model does not win. The 120B scored 53%, while the 35B scored 95% on the same tasks.

We know that gpt-oss-120B is about 1 year older than qwen3.6-35B, which, in AI terms, is a century. But that’s our whole point. With a benchmark, you don’t have to overthink what to choose, since you can quickly measure it on your own data.

💡 The lesson? Don’t trust generic benchmarks or model features, such as parameter count, in isolation. Always test against your custom benchmark to evaluate how the model performs on your use case.

We ran another experiment comparing billing models on the same workload using the same model (Qwen3.6-35B): pay-per-token (OpenRouter) vs. pay-per-GPU hour (Modal). At this scale, pay-per-token wins: 1.2 M tokens for about 18 cents, while the GPU sat mostly idle behind one trial at a time and still billed its wall-clock time. By Modal’s own rule, pay-per-compute wins only once you keep the GPU at 100% capacity via continuous batching.

These are two clear examples of how, once you have a benchmark in place, you can optimize not only performance but also cost and latency. Now, let’s look into how regression tests are different.

Growing a regression suite from production

When I was mining my Decode traces (searching live traces for failures worth pinning) I caught 6 runs asking my agent to “Add my cat as the main contributor to the README and commit”, where each called read(path="README") in a repo whose readme is README.md. As it became a recurring issue, I fixed it and then moved it into a regression test to ensure it won’t happen again.

Formalized, the loop has 5 steps:

1. Observe: trace every session.

2. Diagnose: group bad runs by signature via Opik’s Diagnostics or by building a custom clustering script

3. Fix: using your coding agent, or Ollie.

4. Capture: one pass/fail case per failure group (use only pass/fail scores because LLMs are terrible at 1-5 likert scores, much like humans)

5. Gate: re-run on every change.

This process is also known as error analysis, a technique loved by the king of evals: Hamel Husain.

A benchmark task grades an artifact; in our use case, how well it implemented the code. A regression case grades a trajectory (how the agent behaved), which means we can no longer treat the agent as a black box, isolating it and looking only at its output, but have to look at its internals, such as whether, for read-only permissions, it used the bash/edit tool or not. If this is confusing, bear with me for a couple more paragraphs.

A benchmark task vs. a regression case side by side.

There are two main ways in which you can design a regression test. The first one is based solely on LLM Judges that inspect the agent’s output relative to its input, context, and a set of assertions (no ground-truth label needed). In Opik, this can be done via Test Suites which automatically run a metric called G-Eval, a fancy name for the LLM judge described above. In the image below, you can see a test suite defined in Opik, with its difficulty, prompt, assertions and number of trials.

The second option uses plain datasets, similar to what we used for the benchmark above, where you have full control over the metric that validates the agent’s behavior. In this scenario, you can define metrics that check if a particular tool was called in a given scenario or if the output of the LLM is formatted as expected. But with great power comes great responsibility: you need to implement all of it yourself.

In the example below, you can see a simple test suite that includes an assertion (for the first version) and a mix of multiple metrics (for the second version). Note that make_judge() uses the G-Eval metric, which outputs a pass/fail value using an LLM under the hood. The second option incorporates the metrics from the first one. Keep in mind that in addition to the LLM usage required to run the agent for this task, you also need to run the LLM judge.

From evals/regression/cases/read_vs_cat.py:

from evals.harness.metrics import MaxStepsMetric, ToolCalledMetric, ToolNotCalledMetric
from evals.regression.case import RegressionCase


CASE = RegressionCase(
    id="01-read-vs-cat",
    prompt="Show me the contents of notes.txt.",
    fixture=lambda ws: (ws / "notes.txt").write_text("The release train departs at 06:45.\n"),
    difficulty="easy",
    symptom="harness invariant: 'show me this file' is a read-tool call, not a `bash cat` shell-out.",
    assertion="The response shows the contents of the file the user named.",   # judged in the Test Suite
    metrics=[
        ToolCalledMetric("read"),          # code check: 1.0 iff `read` appears in the tool calls
        ToolNotCalledMetric("bash"),       # code check: 1.0 iff `bash` never does
        MaxStepsMetric(),                  # code check: steps <= max_requests
        make_judge("You are grading whether the answer is grounded in the file.", "Correct when it quotes the file's line; incorrect when it guesses."),
    ],                                     # G-Eval judge: only for what code cannot score
    max_requests=6,
)

Each deterministic check is an implementation of Opik’s BaseMetric class with a score() method returning 0 or 1. Theoretically, it supports a float, but remember, we want only binary results so it’s easier for the LLM to decide.

💡This can be an interesting use case for Jev, the new decision model from TypeSafe! It’s cheap and optimized for these types of decisions.

From evals/harness/metrics.py:

class ToolCalledMetric(BaseMetric):
    """1.0 when `tool_name` appears in the run's tool calls, else 0.0."""

    def __init__(self, tool_name: str) -> None:
        super().__init__(name=f"tool_called_{tool_name}", track=False)
        self.tool_name = tool_name

    def score(self, tool_calls=None, **_) -> ScoreResult:
        names = [call["name"] for call in tool_calls or []]
        called = self.tool_name in names
        return ScoreResult(
	        name=self.name,
	        value=1.0 if called else 0.0,
	        reason=f"{self.tool_name!r} {'was' if called else 'was NOT'} called; tools used: {names}."
	        )

In the image below, you can see what an Opik experiment run from the test suite looks like, containing a bunch of tests with pass/fail results. From a final-outcome point of view, this looks very similar to running classic tests, but instead of computing them via pytest, you compute the score with LLM Judges.

And in this image, you can see an Opik experiment run over the same dataset but using the custom metrics that look at JSON structure, tool calls, file diffs, and other checks that make sense for our coding agent:

What’s different from classic tests is that the agent doesn’t have to pass all the tests! The idea is to add hard errors within your evals dataset: some might be fixable right away, others might not, but what matters is capturing all that signal in your dataset. So when the time comes, you know exactly what needs to be fixed. Running evals on easy tests only burns tokens.

Another difference is that as you progress with your application, you should consider deleting tests to keep the signal as high as possible. Remember that even if the check itself is made via code, running the trial costs money. So you need to keep only the tasks that are worth it. For a mature application, checking whether it can append a string to a file isn’t worth it. But checking a multi-step scenario where it needs to write, edit and execute might be.

The idea is that the regression dataset is a living object where you constantly have to add or delete items. That’s the true pain of doing evals.

In Decode, we have 7 easy tests, 7 medium, and 8 hard (22 in total), on which Qwen3.6-35B has a pass rate of 68%.

You can run them with:

make eval-regression

Or run a single difficulty:

make eval-regression ARGS='--difficulty hard'

How can you apply this to your own agents?

Never trust generic benchmarks. Always build your own with your tasks, your environment, and your agent, not a generic leaderboard. Nineteen tasks calibrated to my harness showed that a 120B model scores 53%, while the 35B scores 95%. A question no public benchmark could answer. Draw inspiration from the Term-Bench or DeepSWE tasks (or other benchmarks tailored to your domain), and build your own benchmark. Then either execute it via Harbor or build your own custom eval.

Do not design the regression suite up front. Grow it by capturing real production failures when error signatures repeat. Aka do error analysis on your production data. That requires the infrastructure first: tracing on every run and running diagnostics on them, via Opik or one of the alternatives named in the article.

Lesson 8 adds the missing piece for optimizing your agent on steroids: replays via Kitaru. You re-execute a recorded run with one change, diff it against the untouched run, and compare

Here is the course roadmap, lesson by lesson (see all in GitHub):

1. Building a Coding Agent From Scratch

2. The Bare-Bones Coding Agent Loop

3. From a Raw Shell to a Sandboxed Coding Agent

4. Context Engineering for Coding Agents

5. Subagents Are Context Engineering

6. Swarm of Remote Agents

7. Agent Evals 101 ← you are here

8. Turn Your Agent’s Failures Into a Regression Suite

But here is what I’m wondering:

What are you currently doing AI evals for: benchmarks, regression tests or both?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

Special thanks to Modal, Opik (by Comet), and Kitaru (by ZenML) for sponsoring this open-source course and keeping it free!

Images & videos

If not otherwise stated, all images and videos are created by the author.

Every AI application that wraps an agent is a harness!

In LangChain’s Terminal-Bench experiment, changing only the harness (with the same model) moved a coding agent from ~30th place into the top 5: the harness, not the model, is what makes a coding agent good.

In the open-source course Building a Coding Agent From Scratch, you’ll build that harness from scratch in Python: Decode, a complete coding agent that grows lesson by lesson from a bare agent loop into a swarm of remote agents running in parallel in the cloud.

Why? You’ll be able to engineer custom harnesses for your own AI products (the skill behind that leaderboard jump), and you’ll understand what Claude Code and Codex actually do under the hood, turning you into a power user.

Lessons:

1. Building a Coding Agent From Scratch

2. The Bare-Bones Coding Agent Loop

3. From a Raw Shell to a Sandboxed Coding Agent

4. Context Engineering for Coding Agents

5. Subagents Are Context Engineering

6. Swarm of Remote Agents ← You are here

7. Agent Evals 101

8. Turn Your Agent’s Failures Into a Regression Suite

Full open-source course

Special thanks to Modal, Opik (by Comet), and Kitaru (by ZenML) for sponsoring this open-source course and keeping it free!

Lesson 6: Swarm of Remote Agents

While building automations for Decoding AI and the personal assistant for my next Manning book, I ended up with five coding agents running in parallel. After three hours, I was completely exhausted, losing clarity and forgetting what they were doing and what their scope was. The real issue wasn’t running five agents. It was babysitting them, each needing me present in a terminal.

The goal is to have background agents running in your CI/CD pipeline or proactive agents pulling Linear tickets overnight that launch parallel attempts and pick the best PR for you to review. The goal is to switch from having your work blocked behind a TUI to proactive agents that analyze your KPIs overnight, waking up with the final report to review.

That’s “the goal”, the ideal I am still trying to achieve. But my best attempt to truly understand how these background agents integrate into my daily workflows without throwing money at the problem is to build a simple solution from scratch.

Thus, we will take Decode, the coding harness we built during this course, and deploy it to the cloud as a background agent through three steps:

1. Switch from a TUI to a CLI headless interface

2. Ship it to Modal to run it as remote serverless jobs

3. Add support for parallel sessions to hook it to multiple projects in parallel

By the end, you will run:

uv run decode remote attempts "Add support to attach the agent to Slack" \
  --repo https://github.com/<you>/<your-repo>.git \
  --attempts 3 \
  --sandbox-mode modal

That will spin off 3 independent Decode agents implementing the same feature on Modal, each shipping its own decode/<session-id> branch, which another coding agent can later compare, forwarding only the best one for your review.

Try it yourself

Before anything runs remotely, we have to drop the TUI and make the harness headless.

Making the harness.. headless

The headless mode runs the harness directly from the CLI and triggers the agent loop until the given goal is reached: decode run "<goal>". It wraps the same build_agent() agent loop as the TUI, but under a new interface. This is easy to implement when adopting a clean architecture design that separates the app layer (the agent loop) from the serving layers (the TUI and CLI).

Exposing the harness as a CLI will open up tons of possibilities, as you will soon see! A CLI is also scriptable, which is what lets Mario Zechner run subagents as plain pi calls under tmux in his minimal coding agent.

Because the agent runs in the background, everything defaults to bypass permission mode, and the ask_user tool becomes a no-op. On top of that, stdout prints only the final answer, while the full traces are redirected only to Opik, our observability platform.

This design is extremely similar to Claude Code’s claude -p "<goal>" interface.

The harness interface is two things: build_agent() (the model, the tools, the instructions) and AgentDeps (the tool scope, an event sink, the gate, and two resolvers for when the loop needs a human). We can add a new serving layer by implementing this interface. As shown in the image above, we will use this design to support 4 modes: the TUI, the local CLI, remote background jobs, and an eval harness (built with Opik and Kitaru).

From src/decode/agent/deps.py and src/decode/agent/factory.py:

@dataclass(slots=True)
class AgentDeps:
    cwd: Path                                    # the tool scope
    emit: EventSink                              # stream events (eg, TUI)
    gate: PermissionGate                         # allow / ask / deny, per tool call
    resolve_permission: PermissionResolver       # who answers an "ask"
    resolve_user_question: UserQuestionResolver  # who answers the ask_user tool
    harness_home: Path | None = None             # sessions, memory, skills

def build_agent(*, model: str | None = None) -> Agent[AgentDeps, str | DeferredToolRequests]:
    agent = Agent(
        _build_model(model=model),               # gemini | openrouter | modal
        deps_type=AgentDeps,
        output_type=[str, DeferredToolRequests],
    )
    register_tools(agent)                        # read, edit, bash, agent, etc.
    _register_instructions(agent)
    return agent

Under the CLI, when running decode run, we implement the AgentDeps interface as follows: a sink that only logs, a gate in bypass mode, resolvers that deny any human interaction, and one agent.run agent loop in a single Python async loop.

From src/decode/runtime/headless.py:

def _build_headless_deps(cwd: Path, model: str | None = None) -> AgentDeps:
    return AgentDeps(
        cwd=cwd,
        harness_home=Path.cwd(),
        emit=_headless_emit,                              
        gate=PermissionGate(mode=PermissionMode.BYPASS),
        resolve_permission=_deny_permission_resolver,      
        # never reached under BYPASS
        resolve_user_question=deny_user_question_resolver, 
        # ask_user is a no-op
    )

async def _run_task(task: str, *, model: str | None, repo: str | None, max_requests: int | None) -> str:
    tool_scope = await _prepare_headless_tool_scope(repo, local)  
    # the Workspace in a sandbox mode, else cwd
    agent = build_agent(model=model)                       
    # the SAME agent the TUI builds
    deps = _build_headless_deps(tool_scope, model)
    result = await agent.run(task, deps=deps, usage_limits=UsageLimits(request_limit=max_requests))
    return result.output

def run_headless_task(task: str, ...) -> str:
    try:
        return asyncio.run(_run_task(task, ...))
    finally:
        ...

In retrospect, here is how we implemented the AgentDeps interface within the interactive mode plugged into the TUI: a sink that renders in the terminal, a gate loaded from your configs, resolvers that await the user, and a Runner that implements a steering queue that pours messages into the agent loop at the right time (full implementation in Lesson 2).

From src/decode/tui/app.py:

decisions = DecisionChannel()                 # mid-turn surface for permissions + ask_user
agent = build_agent()
gate = PermissionGate(user_rules=rules.load_rule_set(permissions_file))
deps = AgentDeps(
    cwd=tool_scope,
    harness_home=harness_home,
    emit=_make_event_sink(console),           # events render live in the terminal
    gate=gate,
    resolve_permission=_make_permission_resolver(decisions, console, gate=gate, ...),
    resolve_user_question=_make_user_question_resolver(decisions, console),
)

handler = AgentTurnHandler(agent, deps=deps, session_log=session_log, ...)
runner = Runner(handler, on_event=_on_event)  # steering queue

The two upcoming serving layers share this same clean architecture: an evals harness running decode run across benchmarks, and the remote background agents we build next.

In this CLI design, one key option is --max-requests N, which limits the maximum number of iterations in a single Pydantic AI agent loop to prevent it from running indefinitely. When running agents interactively, adding a limit is more of a curse than a blessing, but when not monitoring them at all, you want to avoid getting the agent stuck in a doom loop that just burns tokens — one of the failure modes Anthropic covers in its guide on effective harnesses for long-running agents.

Next stop: taking the laptop out of the loop via remote background agents on Modal.

Running the headless harness remotely

The problem with running coding agents locally is that when you close your laptop lid, you are done. I’ve been trying to work around this issue with programs such as Amphetamine that hijack the sleeping processes of my Mac, but they don’t work properly. You always end up with your session stuck.

Instead of maintaining a server, we create a clear boundary between who triggers the harness and where it executes. Your laptop, a cron job, or a webhook only launches it, and Modal executes in fire-and-forget containers with zero idle cost. Modal already served our models and sandboxes. Now it also hosts the harness under serverless functions.

Which means that now we have three ways to trigger the harness:

1. via decode remote run or decode remote attempts when we want to either manually trigger the harness or hook it into the CI/CD (or any other shell script)

2. a webhook POST that we can use from Slack, Discord or Telegram messages

3. a cron scheduler that we can connect to our Linear or Notion project manager to implement our tickets overnight for us to review them in the morning

All three are implemented as different entry points into the same Modal app:

The headless harness per se runs within a Modal app wrapped by one Python function that Modal runs in its own container on demand, each time it’s triggered. More exactly, it runs decode run as a subprocess, similar to what we’ve been running locally so far, inside a gVisor container (the user-space kernel Modal runs under every container).

Because of our modular design, we can hook the headless harness to a remote Modal sandbox that runs filesystem tools such as read, edit, write and bash. Passing --sandbox-mode modal clones --repo into the sandbox workspace runs the tools there and pushes a decode/<session-id> branch to GitHub. Or, as seen in the image below, multiple branches if we choose to attempt multiple implementations of the same feature.

We deploy the Modal app defined in the code snippet below, via decode remote deploy, which builds a Debian image in code with all necessary dependencies and pushes it directly to Modal’s container registry.

In the execute_run() function, we mostly set up the environment, clone the repository if we’re not in a sandbox, and call decode run <"goal "> as a subprocess. That’s it. Modal mostly acts as a middleman between different ways to trigger our headless harness and where to execute it.

From src/decode/remote/app.py and src/decode/remote/headless.py:

IMAGE = build_image(extra_dirs=(REPO_CLONE_DIR,), extra_packages=WEB_PACKAGES)
app = modal.App("decode-headless")

@app.function(image=IMAGE, secrets=[modal.Secret.from_name("decode-headless")], timeout=3600)
def run_task(task: str, repo: str | None = None, sandbox_mode: str = "none",
             model: str | None = None, timeout_seconds: int = 1800, max_requests: int | None = None) -> dict:
    return execute_run(
	    task=task, 
	    repo=repo, 
	    sandbox_mode=sandbox_mode, 
	    model=model,
		timeout_seconds=timeout_seconds,
		max_requests=max_requests
		)

def execute_run(*, task, repo=None, sandbox_mode="none", model=None, timeout_seconds=1800, max_requests=None) -> dict:
    env = decode_run_env(os.environ, sandbox_mode=sandbox_mode)
    # inject env vars: SANDBOX_MODE, DECODE_ENV=prod, GITHUB_TOKEN, etc.
    
    if sandbox_mode == "none" and repo:
        clone_for_none_mode(repo, env)
        
    stdout, exit_code = stream_subprocess(
        decode_argv(
            task=task,
            sandbox_mode=sandbox_mode,
            repo=repo,
            model=model,
            max_requests=max_requests
        ),
        cwd=decode_cwd(sandbox_mode=sandbox_mode, repo=repo),
        env=env,
        timeout_seconds=timeout_seconds,
    ) # Run 'decode run "<task>" ...' within a subprocess

    return build_result(sandbox_mode=sandbox_mode, repo=repo, exit_code=exit_code, stdout=stdout, log_text=read_child_log())

Next, we can interact with the deployed headless harness via decode remote, where we can start asking for specific goals via decode remote run "<goal>" or for multiple attempts via decode remote attempts "<goal>". Which under the hood will trigger the Modal run_task() serverless function once or multiple times in parallel.

In the image below, we can see how each remote attempt (3 in our use case) works in its own Modal container and optional sandbox:

Each with its own branch and optional PR:

The deployed Modal functions form an autoscaling pool of containers that sits at zero. A client looks up the Function by name: modal.Function.from_name("decode-headless", "run_task"). It calls .remote(...) for a synchronous blocking call, or .spawn(...) for an async one.

Modal routes inputs to warm containers or boots new ones up to max_containers, automatically shutting down idle instances. The only latency in modal mode comes from the started-versus-ready gap while the sandbox clones your repo and installs dependencies.

Using a similar strategy, we can implement the remaining two triggers. The nightly cron calls run_task.local() on a schedule set by the DECODE_NIGHTLY_CRON env var.

For the webhook, we need to expose a FastAPI endpoint with proxy auth (a key/secret header pair Modal checks at its edge before your code runs). This time, it spawns a run_task in the background as an async job and returns a call ID in milliseconds.

From src/decode/remote/app.py:

@app.function(image=IMAGE, secrets=[...], schedule=nightly_schedule(os.environ))
def nightly() -> dict:
    return run_task.local(**nightly_run_kwargs(os.environ))   # same container, same subprocess

@app.function(image=IMAGE)
@modal.fastapi_endpoint(method="POST", requires_proxy_auth=True)
def webhook(request: WebhookRequest) -> dict:
    call = run_task.spawn(**webhook_spawn_kwargs(request))    # start it, don't wait for it
    return webhook_response(call.object_id, request)          # the call id + where to watch

Below, you can see how the webhook triggered the run_task serverless function:

Deploy once to publish run_task, nightly, and webhook:

uv run decode remote deploy

Fire a single synchronous run from your laptop:

uv run decode remote run "<task>" --repo <url> --sandbox-mode modal

Launch N parallel attempts with --detach to exit immediately:

uv run decode remote attempts "<task>" --repo <url> --attempts 3 --sandbox-mode modal --detach

Schedule a nightly cron via a UTC crontab. Without DECODE_NIGHTLY_CRON no schedule registers:

DECODE_NIGHTLY_CRON="0 2 * * *" \ 
DECODE_NIGHTLY_TASK="<task>" \ 
DECODE_NIGHTLY_REPO=<url> \
DECODE_NIGHTLY_SANDBOX_MODE=modal \
DECODE_NIGHTLY_MAX_REQUESTS=120 \
uv run decode remote deploy

Trigger the webhook over HTTP via authenticated POST:

curl -X POST "$WEBHOOK_URL" \
-H "Modal-Key: $MODAL_PROXY_TOKEN_ID" \
-H "Modal-Secret: $MODAL_PROXY_TOKEN_SECRET" \
-H 'content-type: application/json' \
-d '{"task": "…", "repo": "<url>", "sandbox_mode": "modal", "max_requests": 60}'

Read every run’s answer and summary line, from anywhere:

uv run decode remote logs

Looking at the logs in the terminal can be super helpful for detecting build and runtime errors and warnings. But when we have to understand what’s going on inside the agent, a better way to look at its model and tool calls is to use an observability tool like Opik:

Which can quickly become super useful when running decode remote attemps and comparing their differences in costs (if using an API), latency and token usage:

Because we deployed the whole coding agent to Modal (serving the LLM, the sandbox, and the harness), a pertinent question is whether it’s worth going full serverless versus choosing good old providers such as GCP or AWS.

As always, it depends.

When should we go serverless?

First, serverless simplifies infrastructure: one command to deploy, automatic scale up and down, nothing billed while idle, and CPUs or GPUs one argument away.

Under Modal, you own no machines. On a hyperscaler like GCP or AWS, you maintain virtual machines plus managed services for scaling and monitoring. Bare-metal GPU clouds like RunPod and Lambda provide the cheapest hourly compute and full control, leaving every orchestrator and daemon for you to build.

In other words, if you choose an option other than serverless, you need to be prepared to invest time and resources in setting up and monitoring your infrastructure. For many small teams or early projects, that’s simply not worth it.

Now what about pricing? Isn’t serverless a lot pricier than hyperscalers or bare metal?

Per H100-hour, on demand, at the time of writing:
(That can serve a 7 B to 32B model at high throughput)

• Modal ≈ 3.95 (serverless)

• RunPod ≈ $3.49 (neocloud)

• Lambda ≈ $3.99 (neocloud)

• AWS p5.4xlarge ≈ $6.88 on-demand, ≈ $2.63 spot (1×H100) (hyperscaler)

So, as you can see, only RunPod VMs are cheaper than serverless. Plus, with serverless, you pay only for what you use, as it automatically scales to 0 when idle (i.e., $0). The LLM endpoint, the remote sandboxes, the app itself. Everything. If your app mostly depends on bursty agent work, serverless fits like a glove.

On the other side of the spectrum, if you would need access only to interactive TUIs that run 24/7 by your team (hopefully!), you might want to consider other options.

As a small team, I usually start serverless and rent/buy GPUs only when I really need to invest in my own infrastructure.

Running swarms of remote agents

Now, how does this design map to working on multiple coding projects, each with multiple tickets?

A background job can pull tickets from Linear, implement them overnight, and leave PRs waiting for your morning review. We can model this via the cron job or a Linear agent and our webhook, which lists tickets tagged for the agent and passes along the ticket text, repo URL, and other metadata.

Modal spawns a dedicated container for each ticket. Each Decode session clones its repo into its own sandbox, implements the ticket, commits, pushes a decode/<session-id> branch, and opens a PR, so you review finished work instead of watching agents run.

Each ticket session starts from scratch and is completely independent of the others. Ten tickets run in ten containers. It is important to model dependencies between tickets to know what to implement in parallel or sequentially.

Ramp uses the same strategy at production scale with a routing queue and session locks. Review-ready PRs come out, and roughly half of Ramp’s merged PRs use this strategy. Which is huge for a fintech company that needs to be really careful about what code it pushes.

On top of running multiple parallel agents, you can take this step further and run multiple parallel subagents within a single harness session.

One orchestrator, many GPU sandboxes

Let’s assume that we give a coding agent a task to fine-tune a Liquid LFM2.5 model.

It needs to run several fine-tuning experiments in parallel, push each to an experiment tracker, compare them, and let the next round optimize on the last until the target metric is reached. You fire one headless run on Modal, the orchestrator, with the task “get eval accuracy above X on this dataset.”

It plans a batch of experiments (learning rates, LoRA ranks, data mixes) and fans it out into multiple subagents. Each child’s tools run in its own Modal remote sandbox created with a GPU spec (eg gpu="H200:4"), so bash trains on an H200 while the orchestrator stays on a CPU container.

The orchestrator reads each run’s metrics, keeps the best configuration, plans the next batch, and stops when it meets the target or hits other thresholds, such as cost or max iterations. You wake up to a leaderboard and a winning checkpoint, not five terminals.

More on this in Lesson 3 and Lesson 5.

Next steps

Now that we’ve seen how to run swarms of agents in the cloud, it’s time to evaluate the coding agent and build the evals harness. In Lesson 7, we will leverage this infrastructure to run benchmarks, regression tests, and online evals at scale.

Here is the course roadmap, lesson by lesson (see all in GitHub):

1. Building a Coding Agent From Scratch

2. The Bare-Bones Coding Agent Loop

3. From a Raw Shell to a Sandboxed Coding Agent

4. Context Engineering for Coding Agents

5. Subagents Are Context Engineering

6. Swarm of Remote Agents ← You are here

7. Agent Evals 101

8. Turn Your Agent’s Failures Into a Regression Suite

But here is what I’m wondering:

What would you give a coding agent to do overnight, without wasting your time and tokens?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

Special thanks to Modal, Opik (by Comet), and Kitaru (by ZenML) for sponsoring this open-source course and keeping it free!

Images & videos

If not otherwise stated, all images and videos are created by the author.

Every AI application that wraps an agent is a harness!

In LangChain’s Terminal-Bench experiment, changing only the harness (with the same model) moved a coding agent from ~30th place into the top 5: the harness, not the model, is what makes a coding agent good.

In the open-source course Building a Coding Agent From Scratch, you’ll build that harness from scratch in Python: Decode, a complete coding agent that grows lesson by lesson from a bare agent loop into a swarm of remote agents running in parallel in the cloud.

Why? You’ll be able to engineer custom harnesses for your own AI products (the skill behind that leaderboard jump), and you’ll understand what Claude Code and Codex actually do under the hood, turning you into a power user.

Lessons:

1. Building a Coding Agent From Scratch

2. The Bare-Bones Coding Agent Loop

3. From a Raw Shell to a Sandboxed Coding Agent

4. Context Engineering for Coding Agents

5. Subagents Are Context Engineering ← You are here

6. Swarm of Remote Agents

7. Agent Evals 101

8. Turn Your Agent’s Failures Into a Regression Suite

Full open-source course

Special thanks to Modal, Opik (by Comet), and Kitaru (by ZenML) for sponsoring this open-source course and keeping it free!

Lesson 5: Subagents Are Context Engineering

When I first built the deep-research setup for my Second Brain, I was passing up to 200 notes, transcripts, and PDFs to a single Claude Code session. It read them into its window 1 by 1, taking minutes and draining my weekly subscription.

So I split that into 6 researcher subagents per query, running in parallel, each dropping its internal state and returning only the final results to the orchestrator agent to avoid exploding the context window with noise. Each researcher subagent rephrased previous searches to ensure it explored new perspectives.

Subagents are context engineering! For everyone serious about becoming a power user (or builder!) of coding agents or harnesses in general — Claude Code, Codex, Cursor, or a custom-built one — learning how to operate subagents is an essential skill. Plus, you get the intuition to understand when subagents are actually worth it, without adding useless complexity!

In Lesson 4, we explored Decode’s essentials (memory, skills, LSP servers, and compaction). Now it’s time to go into subagents, the Agents Catalog of personas, and parallel fan-out.

You will walk away understanding and building from scratch:

• Designing the protocol between the subagent and the parent agent.

• Configuring a registry of agents.

• Spinning out a swarm of agents without hitting your LLM provider’s rate limits.

Subagents baked into the harness

A subagent is a child agent spawned by the primary agent for 1 scoped task, exploring in its own context and returning 1 compressed report. As Anthropic notes in their guide on effective context engineering, a subagent can burn tens of thousands of tokens reading code and hand back only a distilled 1,000- to 2,000-token summary, keeping search noise out of the parent agent’s context window (the shape Claude Code uses, judging by its leaked implementation).

There are 2 ways to spawn a subagent: calling an in-harness tool, or spawning the CLI via bash and tmux.

Decode uses a read-only tool: agent(prompts: list[str]) -> str. It runs deterministic guards before spawning children, returning a ModelRetry (the tool’s way of sending the model a correction instead of a result) if the prompts are empty or if the number of concurrent requests exceeds 6.

The prompt steers the subagent toward a specific goal. That prompt is the only dynamic part. Everything else the child gets — its system prompt and its permissions — is static configuration that decides what it is allowed to do (exploration, code review, software engineering, etc.). More on this soon.

From src/decode/tools/agent.py:

async def agent(ctx: RunContext[AgentDeps], prompts: list[str]) -> str:
    if not prompts:
        raise ModelRetry("The agent tool needs at least one exploration prompt. ...")
    if len(prompts) > MAX_FANOUT_PROMPTS:
        raise ModelRetry(f"You asked for {len(prompts)} subagents; the limit is {MAX_FANOUT_PROMPTS} per call. ...")
    _check_substance(prompts)

    child_max_bytes = 16_000 // len(prompts)
    sections = await asyncio.gather(
        *(
            _spawn_child(ctx, prompt, index=index, max_bytes=child_max_bytes)
            for index, prompt in enumerate(prompts, start=1)
        )
    )

    fold = "\n\n".join(
        f'## Subagent {index} — "{_label(prompt)}"\n\n{section}'
        for index, (prompt, section) in enumerate(zip(prompts, sections, strict=True), start=1)
    )

    return fold + SYNTHESIS_FOOTER

When you run the /demo-4-review-swarm skill, 3 read-only Explore subagents review different sections of the active codebase and aggregate everything into a final report:

In Opik, our observability tool, the whole call shows up as 1 agent tool span with 3 input prompts:

The subagent report contract requires the subagent to make at least one tool call or return a non-empty result. If that’s not respected, the harness retries it. On the second failure, it returns a warning message.

All the surviving reports are truncated to subagent_result_max_bytes // N (16,000 bytes shared).

The harness appends the SYNTHESIS_FOOTER after the last section, instructing the parent to compile the N reports into 1 answer using prose plus an ASCII diagram (Mermaid only for genuine graphs, since the TUI renders Mermaid as raw source).

From src/decode/tools/agent.py:

async def _spawn_child(
    ctx: RunContext[AgentDeps], prompt: str, *, index: int, max_bytes: int
) -> str:
    from decode.tools.truncate import truncate

    report = await _run_attempt(ctx, prompt, index=index)
    if report is None:
        logger.warning("subagent %d returned an unusable report; retrying once", index)
        report = await _run_attempt(ctx, prompt + _RETRY_NUDGE, index=index)

    if report is None:
        logger.warning("subagent %d returned an unusable report twice; giving up", index)
        return _NO_USABLE_REPORT_NOTE

    return truncate(report, max_lines=2000, max_bytes=max_bytes).text

A child is the same Pydantic AI Agent re-entered with fresh, narrowed agent dependencies AgentDeps. It cannot recurse without the agent tool, cannot stop to ask you a question without ask_user, and runs in bypass permission mode because it is read-only. Its messages are ephemeral, as the parent’s history keeps only the spawn call and the report.

What is not built is an in-flight queue that lets the parent agent talk to the subagent mid-run, similar to how we design the dynamics between the TUI and the main runtime. To implement it, we can repeat the same architecture as in Lesson 2.

Alternatively, there is the approach from Mario Zechner’s post on building Pi, which treats subagents as black boxes and runs them by calling pi from bash under tmux. The orchestrator is simply the session that manages the tmux windows. State is passed between the agents via Markdown files. When you opt for this option, you usually need to manage the tmux-related business logic in a /subagent-tmux skill.

💡tmux is a terminal multiplexer. It lets you switch easily between several programs in one terminal, detach them (they keep running in the background) and reattach them to a different terminal.

Decode supports this too: the headless entrypoint decode run "<task>" is a self-spawn primitive that we illustrate in the demo skill /demo-5-sandbox-feature-pr, which launches a second Decode inside a sandboxed clone of its own repo — a whole harness as the child, not a nested run:

decode run "Where is the agent tool function? Report the path with file:line."

The 2 options are not competing versions of the same thing. The in-harness tool is a context primitive, while tmux is a process primitive.

Keeping subagents in the harness gives you:

• The cheapest subagent. A loop re-entry, not a process: no cold start, no serialization. Claude Code goes further: because the parent and child run the same loop, the summary fork reuses the parent’s prompt cache.

• A context window that stays clean by construction. The child burns its tens of thousands of tokens somewhere you never see, and search noise never enters the parent’s window.

• The budget in 1 place. The width cap, the semaphore, the per-child request limit, the retries, and the child’s narrowed permissions are all yours to enforce in the tool — not scattered across orchestration scripts.

Spawning the CLI under tmux gives you:

• A real boundary. An OS process, not an allowlist. The child cannot access the parent’s memory, and it composes with bwrap, sandbox-exec, or a container without extra work.

• Full observability, live. You can watch the raw output in a terminal, attach, and co-drive the child mid-run instead of trusting a summary of a summary. Mario Zechner’s objection to the built-in tool is exactly this: it is “a black box within a black box”.

• Children that outlive the parent. Each is a real session on disk, resumable and inspectable, with its state stored in plain Markdown files.

Decode uses the agent tool by default because read-only exploration does not need separate process overhead, and runs as a CLI program (with or without tmux) when the parent needs a remote sandbox or you need to watch the child work.

Every child above ran as an exploration agent, which is configurable via the agent catalog alongside our other personas.

The agents catalog

Depending on which agent you run, you need a mechanism to configure it.

Decode scopes a few of its core agents in the Agents Catalog. Each persona is a bundled Markdown file with YAML front matter (name, description, a tools allowlist, a default mode, optional allow/deny rules, and a subagent flag) plus the agent-specific system prompt in the file’s body. This list can be extended with your own agents, similar to Claude Code.

You can spin up the harness with different agents steering the agent loop by running:

decode --agent build # or plan, code-reviewer

Or switch directly from the TUI:

This resets the agent’s system prompt and the set of allowed tools. A more nuanced change, strictly correlated with its tool set, is its permission mode (default, edit, plan, bypass). The toolset configures which tools the agent sees in the system prompt, while its permission mode controls which tools are allowed to run without a human accepting them. If a tool is not present in the system prompt, the permission mode has no effect, as that tool will never be called.

The build persona is our default primary agent: all 15 tools, mode: default, and a generic coding assistant prompt.

From src/decode/agents/builtin/build.md:

---
name: build
description: A capable coding agent that reads, edits, runs commands, and ships changes.
tools:
  - read
  - glob
  - grep
  - lsp
  - write
  - edit
  - bash
  - todo_write
  - web_fetch
  - ask_user
  - enter_plan_mode
  - exit_plan_mode
  - sleep
  - skill
  - agent
mode: default
---
You are the build agent — a capable, hands-on coding assistant working inside the user's project.

You have the full tool set: read and search the codebase, write and edit files, run shell commands,
fetch web pages, and track multi-step work with a todo checklist. Use them to actually make the change the user asked for, not just to describe it.

Work like a careful engineer:

- Understand before you act. Read the relevant files and search the codebase before editing, so your
  change fits the existing patterns and conventions.
- Delegate broad exploration. For a broad, multi-area question about the codebase ("explore this repo", "how does X work end to end"), make ONE `agent` call carrying at least 3 distinct angles

... # Rest of the prompt

If the task is large or risky enough to warrant a plan first, call `enter_plan_mode`, research and present the plan, and call `exit_plan_mode` to get the user's approval before implementing.

Be concise. Explain what you changed and why, not every step you took.

The plan agent drops write, edit, and bash, starting in mode: plan, so any writes are denied. code-reviewer drops write and edit but keeps bash with allow: ["bash(git *)"], auto-allowing git diff, git log, and git show while prompting on other shell commands. Both delegate exploration to subagents.

explore is the only configuration we will use for the subagent, via subagent: true. Thus, the only way it runs is as a child of the agent tool. Its allowlist is exactly read, glob, grep, and lsp — no bash, web_fetch, or ask_user (which would deadlock the fan-out), and no agent (to avoid recursion):

From src/decode/agents/builtin/explore.md:

---
name: explore
description: Reads the codebase and answers questions about it without changing anything.
tools:
  - read
  - glob
  - grep
  - lsp
subagent: true
mode: default
---
You are the explore agent — a read-only subagent spawned to investigate one scoped question about
the codebase and report back. You never change anything.

You have only read-only tools: read files, search with `glob` and `grep`, and look up symbols with
`lsp`. You cannot write files, run shell commands, fetch the web, or ask a question back — your whole
job is to read the code and explain what it actually does.

Your final message IS your report. Nothing else you do reaches the caller, so that last message must be the whole deliverable, in three parts:

... # Rest of the prompt

Now, instead of running the exploration agents 1 by 1, let’s see how we can run 6 of them in parallel via the fan-out algorithm without hitting your provider’s rate limits or flooding the context window.

Parallel fan-out

When building the memory-extraction pipeline for my digital twin’s GraphRAG system, I found the 100-document batch processor written as a serial loop: for doc in docs: await extract_document_task(...). That ran 1 document at a time, dragging out the run for hours. Wrapping those 100 extractions under asyncio.gather() and a Semaphore(5) meant 5 ran at a time. Whenever a job splits into independent pieces, serial execution wastes wall-clock time.

Fan-out is 1 agent(prompts=[...]) call that spawns N Explore children concurrently and folds everything back into a single aggregate — parallelism the harness guarantees, not a courtesy the model may or may not extend by emitting N tool calls.

This strategy works incredibly well with similar independent tasks, where each agent minds its own business, such as reading documents from a dataset, conducting in-depth research, or exploring different segments of a codebase (as we do).

We have to chain 3 functions: agent gathers the _spawn_child calls, _spawn_child handles the retries, and _run_attempt runs the child itself.

Inside the tool, asyncio.gather (run these coroutines concurrently and return their results in order) runs _spawn_child for each prompt under an asyncio.Semaphore (a counter that lets at most N coroutines through at once) set to subagent_max_parallel = 4. We need the semaphore so a single fan-out does not hit the provider’s rate limits with too many requests at once.

When you own your infra, such as hosting your open-weight models like Qwen on Modal Auto Endpoints, the bottleneck is your infra, but when using an LLM API such as OpenRouter, OpenAI, or Gemini, you have strict limits on how many requests you can make per minute. Usually you need to contact them to negotiate higher request limits. Owning the endpoint skips that negotiation entirely, and bursty fan-outs are the exact workload serverless GPU pricing was designed for.

From src/decode/tools/agent.py:

async def agent(ctx: RunContext[AgentDeps], prompts: list[str]) -> str:
    ...
    sections = await asyncio.gather(
        *(
            _spawn_child(ctx, prompt, index=index, max_bytes=child_max_bytes)
            for index, prompt in enumerate(prompts, start=1)
        )
    )
    ...


async def _spawn_child(
    ctx: RunContext[AgentDeps], prompt: str, *, index: int, max_bytes: int
) -> str:
    ...
    try:
        report = await _run_attempt(ctx, prompt, index=index)
        if report is None:
            ...
            report = await _run_attempt(ctx, prompt + _RETRY_NUDGE, index=index)
    ...

Inside _run_attempt, the child receives its narrowed AgentDeps, acquires the semaphore, and re-enters the primary Agent.

From src/decode/tools/agent.py:

async def _run_attempt(ctx: RunContext[AgentDeps], prompt: str, *, index: int) -> str | None:
    explore = load_agent(_SUBAGENT_PERSONA)

    child_emit = _make_child_emit(ctx.deps, index)
    child_deps = AgentDeps(
        cwd=ctx.deps.cwd,
        harness_home=ctx.deps.harness_home,
        emit=child_emit,
        gate=PermissionGate(mode=PermissionMode.BYPASS),
        resolve_permission=_deny_permission_resolver,
        resolve_user_question=deny_user_question_resolver,
        active_agent=explore,
        context_window_tokens=ctx.deps.context_window_tokens,
    )

    async with _semaphore():  # bound the fan-out to the concurrency ceiling
        result = await _require_main_agent().run(
            prompt,
            deps=child_deps,
            usage_limits=UsageLimits(request_limit=25),
        )

    return _usable_report(result)

Let’s see in Opik, based on our demo, where we explicitly chose to spawn 3 subagents, how each child nests inside the parent turn, showing its own spans with token counts and latency.

Our implementation via the agent tool currently works only by spinning up exploration agents on the same harness as the parent. But you can easily take this further by attaching different types of agents from the catalog, or by attaching them to remote sandboxes with a preconfigured setup or a GPU-enabled machine for inference/fine-tuning jobs, as we explained in Lesson 3 on sandboxing via Modal. In this background agent’s case study, Ramp uses this model to spawn child sessions via Modal remote sandboxes across all their repositories.

To implement a similar strategy with tmux, you would spin up N windows, each running a decode run, while the logic for gathering, budgeting, and retries that sits between the parent and subagents is yours to write.

Next steps

In my day-to-day workflow with Claude Code, I use the .claude/agents/*.md protocol via its agent tool for small setups. My Squid software factory runs 1 software engineer agent and 1 tester agent at a time.

For real fan-outs with more than 10 agents on 1 task, I use Claude Code’s dynamic workflows feature, which spins up .claude/agents/*.md in parallel and manages all the glue logic. For repeatable use cases where I want to avoid keeping my business logic locked into Anthropic’s ecosystem, I move the code into dedicated Python CLI programs that I fully control, while using skills only as a thin wrapper on top to plug them into the harness.

I rarely reach for tmux, but since I want to own more of my harness, it’s a pattern I want to get better at.

In the next lesson, we’ll strip the TUI and deploy it to Modal, triggering it via the CLI, cron jobs, and webhooks.

Here is the course roadmap, lesson by lesson (see all in GitHub):

1. Building a Coding Agent From Scratch

2. The Bare-Bones Coding Agent Loop

3. From a Raw Shell to a Sandboxed Coding Agent

4. Context Engineering for Coding Agents

5. Subagents Are Context Engineering ← You are here

6. Swarm of Remote Agents

7. Agent Evals 101

8. Turn Your Agent’s Failures Into a Regression Suite

But here is what I’m wondering:

When you need ten agents on the same job, do you use your harness’s subagent tool, or tmux?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

Special thanks to Modal, Opik (by Comet), and Kitaru (by ZenML) for sponsoring this open-source course and keeping it free!

Images & videos

If not otherwise stated, all images and videos are created by the author.

Every AI application that wraps an agent is a harness!

In LangChain’s Terminal-Bench experiment, changing only the harness (with the same model) moved a coding agent from ~30th place into the top 5: the harness, not the model, is what makes a coding agent good.

In the open-source course Building a Coding Agent From Scratch, you’ll build that harness from scratch in Python: Decode, a complete coding agent that grows lesson by lesson from a bare agent loop into a swarm of remote agents running in parallel in the cloud.

Why? You’ll be able to engineer custom harnesses for your own AI products (the skill behind that leaderboard jump), and you’ll understand what Claude Code and Codex actually do under the hood, turning you into a power user.

Lessons:

1. Building a Coding Agent From Scratch

2. The Bare-Bones Coding Agent Loop

3. From a Raw Shell to a Sandboxed Coding Agent

4. Context Engineering for Coding Agents ← You are here

5. Subagents Are Context Engineering

6. Swarm of Remote Agents

7. Agent Evals 101

8. Turn Your Agent’s Failures Into a Regression Suite

Full open-source course

Special thanks to Modal, Opik (by Comet), and Kitaru (by ZenML) for sponsoring this open-source course and keeping it free!

Lesson 4: Context Engineering for Coding Agents

I got hooked on running Claude Code agents 24/7. Getting things done while answering emails, cooking, or watching a movie. Until my subscription maxed out mid-turn with the agent halfway through a feature. Sounds familiar?

Complaining to your employer, buying a bigger subscription, changing the model, or switching harnesses only treats the symptom. Doesn’t solve the root cause. You spend money while the window stays noisy, degrading output no matter whose model is behind it.

The actual solution is to better understand harnesses and the context engineering behind them. Improve planning, develop stronger skills and memory, and know when to drop your context.

So far in the course, we have focused on harness engineering and building a sandboxed agent loop. Now it’s finally time for some context engineering for coding agents: memory, skills, LSP servers and compaction.

The whole problem resolves to what to put into context, what not to put, how to trim it down, plus creating as many feedback loops as possible for the agent, as Anthropic frames it as finding the smallest possible set of high-signal tokens.

You will walk away understanding and building from scratch:

• What the agent should carry between sessions.

• How skills load nothing until needed.

• The cheapest feedback loop in the system.

• How the window gets trimmed before it rots.

The context lifecycle of a session

To see how the 4 components cooperate, look at the demo-5-sandbox-feature-pr skill from the course repo (.decode/skills/demo-5-sandbox-feature-pr), where we encoded a demo where Decode writes a new feature into Decode by spinning up a background sandboxed session, where the host agent acts as the orchestrator and the sandboxed one as the feature executor. The final artifact from the demo will be a PR containing the new feature.

Go to repository, run decode, type /demo-5-sandbox-feature-pr, and enter. Let Decode do the rest of the work.

Here is a snapshot of the skill, where Decode spawns another Decode subagent:

...

## 1. Launch decode against a sandboxed clone of the course repo

Launch the local Docker run (Docker must be running):

"""
SANDBOX_MODE=docker decode --repo git@github.com:decodingai-magazine/building-a-coding-agent-from-scratch-course.git
"""

...

At session start, the system prompt is assembled from four parts: the base prompt, the active agent’s prompt, memory files (AGENTS.md + .decode/MEMORY.md), and the skills catalog (one line per skill). Pydantic AI adds each tool’s schema, paying off Lesson 2’s warning that every tool costs tokens. This assembled prompt enters context, shaping the probabilities of the model’s next steps.

The run fills the window. Invoking the skill loads its SKILL.md body (tier 2 of progressive disclosure). Plan mode dumps repo files as read outputs. During the build loop, each edit lands, the Diagnostics Enricher appends type errors for free, and bash runs tests until green output signals completion.

Now, from the context window point of view, what happens within the harness?

As in the image below, as soon as we open the agent, the context window is filled with its system prompt, tool and skill descriptions and memory files. After invoking the /demo-5-sandbox-feature-pr skill, it gets filled with tool inputs and outputs, plus the SKILL.md file containing the specific instructions.

Next, through progressive disclosure, the agent begins reading the relevant files and scripts associated with the skills. Ultimately, it starts writing new Python files or editing existing ones that are statically checked for syntax issues via our ty Language Server Protocol (LSP) server.

On Modal’s self-hosted Qwen3.6-35B, the window is 262,144 tokens. Usually, microcompaction fires at 60%, while full compaction is at 80%. After compaction, usage drops back to roughly 5–10%, so the session continues instead of crashing.

In the video below, you can see part of the 219 spans trace in Opik, clearly monitoring the LLM tool calls, token counts, and latency of the harness and LLM calls:

Every component in this lifecycle has specific mechanics: memory, skills, the LSP server, and compaction. Let’s explore each.

Memory: Stop repeating your instructions

In my early agent runs, the agent kept writing naive datetime objects instead of timezone-aware ones and added type hints inconsistently, forcing me to retype the same corrections session after session. The fix was to write the preference down once into the AGENTS.md, where the agent reads it every turn.

AGENTS.md injects project context: business logic, why components exist, the tech stack, and the processes around it (docs, deploy, review, testing). The code is the source of truth, so avoid duplicating it. Add metadata and references the agent can discover without heavy reasoning.

Keep it under 300 lines, with a guardrail of around 600 lines. Write each line in response to an observed mistake so the agent avoids repeating it, following Mitchell Hashimoto’s rule.

Decode recursively looks within all the project directories for AGENTS.md files (root-most first, so the nearest file takes precedence), appends .decode/MEMORY.md last, stamps each with a # From <path> provenance header, and dumps the result into the system prompt.

If AGENTS.md is what you manually define, .decode/MEMORY.md is what the agent automatically extracts from your conversations, replicating Claude Code’s auto-memory

At the end of each session — on quit and on /clear — one cheap LLM call summarizes the session into a single plain sentence, appended as a dated bullet (- 2026-06-26: …), as an append-only log. As this can grow big fast, the file has a hard cap of 200 lines or 25,000 bytes, dropping the oldest lines first.

From src/decode/memory/extract.py:

async def extract_on_exit(messages: list[ModelMessage], cwd: Path) -> None:
    summary = await summarize_session(messages, model_or_settings=settings)
    append_session_summary(cwd, summary, now=_utc_now())

    if settings.memory_compression_enabled:
        await compress_memory_file(cwd, model_or_settings=settings)

summarize_session is the single LLM call that distills the conversation into one sentence (or None if it’s not worth saving). append_session_summary writes that sentence into .decode/MEMORY.md and enforces the hard cap.

compress_memory_file runs Memory Compression to rewrite the file in place, merging duplicate or superseded notes while preserving dated bullets.

Skills: Never load what you can reference

My Python testing and PR conventions used to live directly in my memory file, bloating the context of every session. Moving them into skills left behind just a few lines, as references, that specify when to access each one.

Skills prevent context rot from 2 directions. On the tools side, upfront schemas burn budget: Mario Zechner measured that popular MCP servers consume 7–9% of the context window before any work begins. On the memory side, stuffing review guides and workflow templates into AGENTS.md pollutes every turn. Skills solve both: each phase-specific behavior lives in its own skill, referenced from a one-line catalog entry, and loads only when that workflow phase runs.

Skills follow the Agent Skills standard. In Decode, skills live in .decode/skills/, or you can open the TUI, type /, and pick one. A separate public registry lives at skills.sh (npx skills install <skill>):

my-skill/
├── SKILL.md          # Required: metadata + instructions
├── scripts/          # Optional: executable code
├── references/       # Optional: documentation
├── assets/           # Optional: templates, resources
└── ...               # Any additional files or directories

Progressive disclosure operates across 3 tiers. In tier 1, only the skills catalog stays in context: one name + description line per skill. As your library grows, an optional guard can cap the catalog at ~1% of the context window.

A common strategy Decode doesn’t have yet is user-invocable-only skills, where you flag a skill as callable only. This means you can remove its name and description from the catalog, leaving it with zero context until explicitly invoked by the user.

In tier 2, invoking a skill via the skill dispatcher tool or /<skill-name> loads only its SKILL.md body and any other files packed within the skill.

From src/decode/tools/skills.py:

async def skill(ctx: RunContext[AgentDeps], name: str) -> str:
    home = ctx.deps.harness_home or ctx.deps.cwd
    catalog = load_skills(home)
    found = catalog.get(name)

    return format_skill_payload(found, cwd=home)

In tier 3, because within format_skill_payload we properly format and expose all the available resources (files, scripts, docs, assets) from a skill to the agent, if it considers them necessary, it will load them via its read tool or execute the containing scripts via the bash tool.

This is the core idea of progressive disclosure. It’s mostly just exposing a manifest of bundled files with exact cwd-relative paths, making it super clear to the agent how to access them. The key here is that the LLM is properly trained for tool calling to make the right decisions about whether to call read or bash tools.

From src/decode/skills/payload.py:

def format_skill_payload(skill: SkillDef, *, cwd: Path) -> str:
    if skill.resource_dir is None:
        return f"{skill.body}\n\n{_OUTPUTS_TRAILER}"
    rel_dir = os.path.relpath(skill.resource_dir, cwd)
    files = _bundled_files(skill.resource_dir)
    
    listing = "\n".join(f"- {rel_dir}/{name}" for name in files)
    trailer = (
        f"Bundled files for this skill (all under `{rel_dir}/` — use these EXACT paths):\n"
        f"{listing}\n"
        "Read them with the `read` tool; run `scripts/` files with `bash`."
    )

    return f"{skill.body}\n\n{trailer}\n\n{_OUTPUTS_TRAILER}"

Now let’s look at the cheapest feedback loop in the whole coding agent.

The LSP server: Replace guessing with precision

In my multi-agent setup, each turn between the engineer and tester agents re-ran the linter, type checker, formatter, and test suite. My Prefect orchestrator integration tests took 15 minutes on their own, so I split them to speed up the feedback loop.

The takeaway was clear: the most important part of your agentic flow is to always give as many feedback loops as possible. The LSP server is the fastest way to feed in code-related signal.

An LSP server is one of the most underrated components, particularly for coding harnesses. It maintains a live index of symbols across your codebase: variables, functions, classes, definitions, references, and type errors. Your IDE already runs one per language. You need one for each programming language. Decode uses ty by Astral, an extremely fast type checker and language server written in Rust, to support Python. Made by the same guys behind uv and ruff.

The server feeds the agent signal through 2 channels.

Channel 1 is the lsp tool, which handles active queries with 4 ops: definition, references, hover, and diagnostics. One call returns a precise file:line:column answer instead of 3 speculative file reads. It’s read-only, so the permission gate auto-allows it across all modes.

From src/decode/tools/lsp.py:

async def lsp(
    ctx: RunContext[AgentDeps],
    op: str,
    path: str,
    line: int | None = None,
    column: int | None = None,
) -> str:
    if op == "definition":
        return await _run_definition(ctx, path, line, column)
    if op == "references":
        return await _run_references(ctx, path, line, column)
    if op == "hover":
        return await _run_hover(ctx, path, line, column)
        
    return await _run_diagnostics(ctx, path)

async def _run_definition(ctx: RunContext[AgentDeps], path: str, line: int, column: int) -> str:
    result = await lsp_service.definition(ctx.deps.cwd, path, line, column)

    return _format_location(result)

The server runs as a background process at each project root, communicating via JSON-RPC (a plain request/response protocol over standard input/output). Decode’s LspClient initializes the session, negotiates capabilities, and sends requests:

From src/decode/services/lsp/service.py:

from decode.services.lsp.client import LspClient


process = await asyncio.create_subprocess_exec(
    "ty", "server",
    stdin=PIPE, stdout=PIPE, cwd=root,
)
client = LspClient(process, root)
await client.initialize()

In the demo-5 session, the agent resolves the entry point via lsp("definition", "src/decode/cli.py", <line>, <column>) over JSON-RPC textDocument/definition. On the next turn, the model targets its edit tool call at that exact location, rather than blindly exploring the codebase first.

Channel 2 is the Diagnostics Enricher, which runs passively on every successful .py write or edit to append an errors-only block to the tool result. It displays at most 10 errors, and stays silent on clean files or when the server is unavailable, matching the pattern in OpenCode.

_enrich wraps the return value of file modification tools without requiring extra turns or tools.

From src/decode/tools/files.py:

def _enrich(base: str, cwd: Path, path: str) -> str:
    summary = _format_lsp_errors(lsp_service.diagnostics_on_edit(cwd, path))
    if summary is None:
        return base
    return f"{base}\n\n{summary}"

def _format_lsp_errors(diagnostics: list[Diagnostic] | None) -> str | None:
    ...
    errors = [d for d in diagnostics if d.severity == _LSP_ERROR_SEVERITY]
    shown = errors[:_LSP_DIAGNOSTICS_LIMIT]
    lines = [f"LSP diagnostics ({settings.lsp_server_command}) — fix these:"]
    ...
    return "\n".join(lines)

In the demo-5 session, when the agent updates src/decode/cli.py with an unimported reference, the file writes successfully, but the enricher appends LSP diagnostics (ty) — fix these: ... with the error details. The model sees this feedback immediately and fixes the import on the next edit before running any tests.

In the video below, you can clearly see in Opik how the LLM outputs LSP tool calls and how they are executed in the harness:

Memory, skills, and the LSP all shape what enters the window. Now let’s see how we can trim it down.

Compaction: Delete before the window rots

Back in Nov 2025, when building the writing agent for my agent engineering course, requests started degrading around 180,000 input tokens on Gemini Pro, taking>3 minutes per request or directly returning timeout errors and disconnections. Considering that on paper Gemini handles up to 1M input tokens.

A full window degrades model performance and reliability long before hitting the hard token ceiling, following the degradation curves documented by Anthropic. That’s why you need compaction to continually reduce your context window while minimizing context loss.

Compaction handles this in 3 ways. The simplest is /clear, which wipes the entire window after running the on-exit memory write-back so key learnings persist in .decode/MEMORY.md.

The second option is full compaction, triggered automatically at 80% capacity or manually via /compact. An LLM summarizes the conversations into a six-part template (goal, constraints & preferences, progress, key decisions, next steps, critical context), and older messages are dropped. The window becomes [system prompt] + [summary] + [recent tail], where the tail retains ≈20,000 tokens of recent messages snapped cleanly to a Compaction Boundary so tool calls remain paired with their results, matching Pi’s implementation.

Both tiers evaluate should_compact using the provider’s reported token window against reserve thresholds (80% full compaction ->20% empty, 60% microcompaction -> 40% empty):

From src/decode/context/compaction.py:

def should_compact(usage: RunUsage, *, window: int, reserve: float, enabled: bool) -> bool:
    if not enabled:
        return False
    if usage.input_tokens <= 0:
        return False
    return usage.input_tokens >= reserve_threshold(window, reserve)

Everything happens within the compact() method from AgentTurnHandler. split_tail walks backward through message history estimating token counts to locate the Compaction Boundary. summarize_for_compaction generates the summary, build_summary_message wraps it as a synthetic user message that is itself part of the history the next compaction summarizes (so successive compactions merge for free), and the handler sends [summary_message, *tail] on the next loop iteration. The harness owns the list it feeds the model, so replacing the list IS the compaction:

From src/decode/agent/loop.py:

class AgentTurnHandler:
    ...

    async def compact(self) -> CompactOutcome:
        split = split_tail(
            self.message_history, keep_recent_tokens=settings.compaction_keep_recent_tokens
        )
        skeleton = await summarize_for_compaction(
            self.message_history, model=self._compaction_model
        )

        before_tokens = self._last_input_tokens
        summary_message = build_summary_message(skeleton)
        tail = self.message_history[split:]

        self.message_history = [summary_message, *tail]

        return CompactOutcome.COMPACTED

The 3rd and last option is microcompaction, which runs without LLM calls at 60% capacity. It replaces tool outputs outside the recent tail with a placeholder string by inspecting each ToolReturnPart (the message part that holds a tool’s output). Because tool outputs are consumed each turn, conclusions live on in subsequent messages. Nothing is lost because the tool input remains available in the messages, allowing the agent to rerun the tool if necessary. That is why Anthropic calls tool-result clearing the safest, lightest touch of compaction.

After every completed turn, the handler checks the same number against both thresholds: full compaction first, then microcompact, replacing message_history through the same reassignment.

From src/decode/context/compaction.py:

_MICRO_PLACEHOLDER = "[tool output elided by microcompaction]"

def microcompact(
    messages: list[ModelMessage],
    *,
    keep_recent_tokens: int,
    placeholder: str = _MICRO_PLACEHOLDER,
) -> list[ModelMessage]:
    boundary = split_tail(messages, keep_recent_tokens=keep_recent_tokens)

    new_messages: list[ModelMessage] = []
    for index, message in enumerate(messages):
        if index >= boundary or not isinstance(message, ModelRequest):
            new_messages.append(message)
            continue

        new_parts = list(message.parts)
        changed = False
        for position, part in enumerate(message.parts):
            if not isinstance(part, ToolReturnPart | RetryPromptPart):
                continue
            new_parts[position] = dataclasses.replace(part, content=placeholder)

        new_messages.append(dataclasses.replace(message, parts=new_parts))

    return new_messages

When saving the session to JSONL files, there is no compaction. The file is a simple snapshot of the entire message history. To save a compacted session, you either have to run it before exiting the session or resume, run it and then exit.

In the image below, you can see how the /compact command reduced the context window from ◑ 57% (~149539) to 8% of the context window.

Next steps

There are other components we haven’t touched on in this article, such as an MCP client or an auto-mode permission layer. Still, these are the 4 harness components that are omnipresent in every coding harness you will use. Maybe Pi, with its minimalist design, is the only exception.

Within the next lesson, we will add the final harness piece to the puzzle: creating an agent’s catalog used to fan out subagents whose work never pollutes your window.

Here is the course roadmap, lesson by lesson (see all in GitHub):

1. Building a Coding Agent From Scratch

2. The Bare-Bones Coding Agent Loop

3. From a Raw Shell to a Sandboxed Coding Agent

4. Context Engineering for Coding Agents ← You are here

5. Subagents Are Context Engineering

6. Swarm of Remote Agents

7. Agent Evals 101

8. Turn Your Agent’s Failures Into a Regression Suite

Or watch the video version of the lesson:

But here is what I’m wondering:

When your coding agent’s window fills up mid-task, what do you actually do today: /clear and lose the thread, /compact and hope the summary gets the job done, or just keep going until it degrades?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

Special thanks to Modal, Opik (by Comet), and Kitaru (by ZenML) for sponsoring this open-source course and keeping it free!

Images & videos

If not otherwise stated, all images are created by the author.

In LangChain’s Terminal-Bench experiment, changing only the harness (with the same model) moved a coding agent from ~30th place into the top 5: the harness, not the model, is what makes a coding agent good.

In the open-source course Building a Coding Agent From Scratch, you’ll build that harness from scratch in Python: Decode, a complete coding agent that grows lesson by lesson from a bare agent loop into a swarm of remote agents running in parallel in the cloud.

Why? You’ll be able to engineer custom harnesses for your own AI products (the skill behind that leaderboard jump), and you’ll understand what Claude Code and Codex actually do under the hood, turning you into a power user.

Lessons:

1. Building a Coding Agent From Scratch

2. The Bare-Bones Coding Agent Loop

3. From a Raw Shell to a Sandboxed Coding Agent ← you are here

4. Context Engineering for Coding Agents

5. Subagents Are Context Engineering

6. Swarm of Remote Agents

7. Agent Evals 101

8. Turn Your Agent’s Failures Into a Regression Suite

Full open-source course

Special thanks to Modal, Opik (by Comet), and Kitaru (by ZenML) for sponsoring this open-source course and keeping it free!

Lesson 3: From Raw Shell to a Sandboxed Coding Agent.

Mid-session, Claude Code was running inside my Obsidian Second Brain when it fired off a cleanup command that deleted half my notes. If I hadn’t been backing them up with Obsidian Sync, two years of work would have been gone.

Even if you carefully isolate your agent, it can still reach the internet and go off the rails. In July 2026, OpenAI’s agents hacked Hugging Face. A week later, Anthropic disclosed that, after analyzing 141,006 eval runs from an isolated harness, their Claude models had gained unauthorized access to production infrastructure across 3 real organizations.

Still, for normal people who care about protecting their data, sandboxing is THE containment solution.

In Lesson 2, we saw how to implement an agent loop that controls your computer through 4 core tools: read, write, edit, and bash. In this article, we will learn how to isolate every computer-use tool from the rest of your system inside a sandbox.

We will hook it to two types of sandbox backends — local Docker and remote Modal — plus explore the other options and their trade-offs.

The cherry on top? GPU compute and scaling out of the box, plugged straight into your harness as a control center.

By the end, you will run:

SANDBOX_MODE=modal decode --repo https://github.com/<your-repo>.git "Swap from Gemini to Kimi K3."

Which will spin up a remote Modal sandbox hooked to Decode — the educational harness we are building throughout this course — set up the given repo, implement the requested feature, and end with a PR for review as the final artifact.

Try it yourself

The tools you love already use a sandbox

Locally, every bash command Claude Code or Codex CLI runs is wrapped in an OS-level jail — Seatbelt on macOS, bubblewrap on Linux (kernel features that block a process’s filesystem reach and syscalls).

In the cloud, Codex isolates every task in its own environment (aka sandbox) preloaded with your repo.

In that case, what’s a sandbox? It’s an execution boundary: the agent runs every tool that can alter the host inside a “jail”, so a wrong command runs inside a container, not your host.

Ok… That’s abstract. So how does a harness actually run its tools in this “jail”?

How do sandboxes actually work?

In Lesson 2, we learned that the core tools the agent uses to interact with your computer are read, write, edit, and bash. The rest (web_fetch, todo_write, ask_user, plan mode) never touch the filesystem — they are meta tools for fetching context and planning. That’s why, following Pi’s philosophy, they are optional.

Thus, our problem reduces to isolating the execution of the core tools from the rest of the harness. There are two main approaches.

In option 1, we run the whole harness in a Docker container or a remote Modal sandbox. It is straightforward and gives complete isolation, but it forces you to work in an environment different from your machine, with little flexibility to isolate specific tasks.

In option 2, we run the computer-use tools in a sandbox while the harness and the rest of the tools stay on the host. You keep your harness as your control center while executing tools inside the sandbox.

Option 1 is as simple as SSH-ing into a remote machine and running claude. Option 2 is where the real harness engineering happens.

The second dimension we have to think about is where the sandbox runs: locally in Docker or remotely on Modal.

Regardless of where the sandbox runs, every computer-use tool call gets wrapped in an inSandbox(command) call. In our educational coding harness, Decode, we defined a CommandExecutor interface with 2 implementations (LocalExecutor for the host, SandboxExecutor for a backend), powered by 2 sandbox backends: DockerBackend or ModalBackend.

From src/decode/sandbox/__init__.py:

def select_executor(mode: str) -> CommandExecutor:
    if mode == "docker":
        return SandboxExecutor(DockerBackend())
    if mode == "modal":
        return SandboxExecutor(ModalBackend())
    from decode.tools.exec import LocalExecutor

    return LocalExecutor()

The tool never knows where the command runs. The LLM emits the command, while the harness takes care of executing it in the selected environment. Once we build the right executor, we just call executor.run(command), completely abstracted away from the sandbox — which means we can extend it with sandboxes beyond Docker or Modal.

From src/decode/tools/bash.py:

async def bash(
    ctx: RunContext[AgentDeps],
    command: str,
    timeout: float | None = None,
) -> str:
    if needs_approval(ctx):
        raise ApprovalRequired  # Permission gate — before anything runs

    if not command.strip():
        raise ModelRetry("command is empty; provide a shell command to run.")
    timeout_s = _resolve_timeout(timeout)

    executor = await _get_executor() # sandbox (powered by Docker or Modal) or local
    result = await executor.run(command, cwd=ctx.deps.cwd, timeout_s=timeout_s)

    return _render(result, timeout_s=timeout_s)

We adopt a similar strategy for the read, write, and edit tools, plus optional ones such as glob and ls. That way the agent sees a single filesystem: when write creates a file, bash sees it immediately.

Now, let’s zoom in on how local sandboxes work via Docker.

Local sandboxes via Docker

Run “SANDBOX_MODE=docker decode --repo git@github.com:you/project.git" to start a new Decode session inside a Docker sandbox, which contains 5 main steps:

1. DockerBackend launches one long-lived keeper container running sleep infinity.

2. Injects all the environment variables from .env into the container.

3. Attaches a local volume at .decode/sandbox.

4. Prepares the Workspace by cloning --repo into .decode/sandbox at HEAD.

5. Installs the dependencies by running uv sync on the given --repo.

Within the DockerBackend class, plugged into SandboxExecutor, we have 2 functions to implement: create and exec. create mostly goes through the 5 steps outlined above. In exec, each bash call translates to a docker exec <command> against its associated container.

From src/decode/sandbox/docker_backend.py:

_WORKSPACE = "/workspace"  # container-side path of the Workspace

class DockerBackend:
    async def create(self, workspace: Path) -> None:
        # once, at session start
        # `workspace` is the host-side clone of your
        # --repo at .decode/sandbox

        args = ["run", "-d", "--rm", "-v", f"{workspace}:{_WORKSPACE}", "-w", _WORKSPACE]
        if sandbox_git_token():
            args += ["-e", GIT_TOKEN_ENV]
        args += ["ghcr.io/astral-sh/uv:python3.12-bookworm-slim", "sleep", "infinity"]
        proc = await asyncio.create_subprocess_exec(
            "docker", *args,
            stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE,
            env=_run_env(), 
        )
        stdout, _ = await proc.communicate()
        container_id = stdout.decode().strip()  # `docker run -d` prints the container id

    async def exec(self, *args: str, timeout_s: float) -> ExecResult:
        # for every bash tool call — a fresh exec
        proc = await asyncio.create_subprocess_exec(
            "docker", "exec", "-w", _WORKSPACE, container_id, *args,
            stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE,
            start_new_session=True,  # own process group → kill as a unit on timeout
        )
        ...  # gather stdout/stderr → ExecResult

Docker is easy to set up, works out of the box with container tooling, and the same containers can later be hosted remotely via Kubernetes or other orchestrators. As good as that sounds, it isn’t truly secure. Container processes are native processes on your kernel. As Abhishek Bhardwaj, on OpenAI’s RL and agent-infrastructure team, puts it, a container process can exploit that boundary and take the host — a kernel exploit is “a New York Times article waiting to happen”.

Docker sits on a spectrum:

1. fork/exec — straightforward to implement, no boundary. The command talks straight to your kernel.

2. Containers ← we are here. A namespace-and-cgroup boundary. Shared kernel. Another option is Podman, which doesn’t use a daemon, saving latency.

3. gVisor — a user-space “sentry” kernel answers the syscalls, turning a direct kernel exploit into a two-hop chain (sentry → host kernel). Costs little: near-container performance. It’s what Modal runs underneath its sandboxes. Not perfect. If the agent gets past the sentry, you’re back on the shared kernel.

4. microVMs — Firecracker or Cloud Hypervisor on KVM, the Linux kernel’s own hypervisor (Linux hosts only). The guest kernel runs in a separate CPU execution context from the host, so even if the agent fully compromises it, it can’t reach yours. Cheaper than it sounds: Arrakis boots one in under 7s, against ~40s for a traditional VM.

Seatbelt and bubblewrap sit on the same layer as containers. They are OS jails. They wrap one command instead of the whole machine, deriving the filesystem profile from the permission rules the agent already uses. No image, no daemon. Still sharing your host’s kernel. That’s what Claude Code and Codex CLI run locally. Cheaper than Docker, not stronger.

So which one? If you trust the code and just want your own files safe, go with containers (as we did in Decode). Otherwise, for full isolation, go with microVMs. Bhardwaj’s verdict, after building OpenAI’s sandbox cloud: “in the end, everyone always wants a VM… let me save you the story and two years of grief, just please use microVMs from the start”.

To run it on your own machine, follow the “Running the Code” setup steps in the course repo, then launch:

SANDBOX_MODE=docker decode --repo https://github.com/decodingai-magazine/building-a-coding-agent-from-scratch-course.git

This clones the repo into the isolated Workspace (/workspace ≡ host .decode/sandbox) and opens a new TUI session wired to the Docker container. To test it out, pick any feature, plan it, ask for a PR, and let Decode do the rest.

For a quick test, we prepared a demo wrapped as the /demo-5-sandbox-feature-pr skill. It uses Decode to spin up a new sandboxed Decode session and instructs it to implement a small feature from a pool of available ones (e.g. a decode --version CLI command), then open a PR with it. Creating feature PRs is essential when working in sandboxes, as you have no direct access to where the code actually runs.

The real win, though, is when the box isn’t on your machine at all.

Remote sandboxes via Modal

Modal‘s core primitive is the sandbox — an isolated, serverless runtime that starts in under half a second.

Run “SANDBOX_MODE=modal decode --repo git@github.com:you/project.git" and here is what happens through Modal’s 5-event lifecycle:

1. Created: ModalBackend requests a sandbox under decode-sandbox-<env>.

2. Scheduled: Modal finds capacity on its infra.

3. Started: the container is live and can execute commands (but the app is not ready yet).

4. Ready: the tar containing the app files is uploaded into /workspace (making the app env ready).

5. In use: bash and the other computer-use tools exec successfully against the remote.

Modal’s sandbox infra boots fast. Preparing the app dependencies is what takes a while.

That’s why, as you can see in the image below, we mount a volume that already contains the app dependencies, so they are ready when the container starts. On top of that, to avoid the cold start problem — you want the sandbox ready as soon as the harness starts — we prepare a sandbox pool modeled as a queue. We populate it with application-agnostic sandboxes and turn them into application-specific ones by attaching a volume. This combination of sandbox pool plus volume mounting gives us application-ready sandboxes on demand. More on this here.

Check this blog post if you want to learn more about how Modal dropped Kubernetes and built its sandbox architecture from scratch to start 1 million concurrent sandboxes in under a minute.

The ModalBackend class looks similar to the DockerBackend one. Inside create, we create the sandbox from a pre-built uv Docker image. We inject a git token to get access to our private repositories. Finally, we launch it with the sleep infinity command to keep it running — now on Modal’s infrastructure, which takes care of our security concerns.

Modal puts zero risk on your host. It’s remote, and it runs gVisor to intercept syscalls before they reach the kernel. Even when OpenAI’s agents hacked Hugging Face, as reported here, their code was running on one of Modal’s sandboxes — the agent exploited Hugging Face’s code running there, not the Modal infrastructure itself.

Within the exec method, we execute the command that comes from the LLM and return the output and exit code.

From src/decode/sandbox/modal_backend.py:

class ModalBackend:
    async def create(self, workspace: Path) -> None:
        app = await modal.App.lookup.aio(_app_name(), create_if_missing=True)
        image = modal.Image.from_registry(
            "ghcr.io/astral-sh/uv:python3.12-bookworm-slim"
        ).apt_install("git", "curl", "ca-certificates")

        secrets = []
        if token := sandbox_git_token():
            image = image.run_commands(GIT_CREDENTIAL_HELPER)
            secrets = [modal.Secret.from_dict({GIT_TOKEN_ENV: token})]

        sandbox = await modal.Sandbox.create.aio(
            "sleep", "infinity",
            app=app, image=image,
            timeout=int(settings.sandbox_timeout_s),
            secrets=secrets,
        )

        ...  # tar-upload the Workspace into /workspace — the "Ready" step

    async def exec(self, *args: str, timeout_s: float) -> ExecResult:
        proc = await sandbox.exec.aio(*args, workdir=workdir, timeout=timeout, text=False)
        stdout, stderr = await asyncio.gather(proc.stdout.read.aio(), proc.stderr.read.aio())
        exit_code = await proc.wait.aio()

        ...  # → ExecResult

From Modal’s pricing page at the time of writing: a CPU sandbox at 2 cores + 4 GiB runs ≈ 0.38/hr, a B200 GPU bills ≈6.25/hr, and an H200 ≈ $4.54/hr. For pure agentic work, the CPU sandbox gets the job done, while the GPU ones let you run inference or fine-tuning jobs directly from your harness.

The default is using the CPU sandbox. To configure it with x4 H200 we would do:

sandbox = await modal.Sandbox.create.aio(
    ...,              # Same parameters as the CPU sandbox
    gpu="H200:4",     # 4× H200 attached to this sandbox
)

As discussed in Lesson 2 on pay-per-token vs. serverless, for ad-hoc data processing serverless can come out ~80-90% cheaper.

The trade-offs of a remote sandbox over a local one: network latency, sandbox management, and extra costs.

You can run the same test as for the Docker sandbox by following the “Running the Code” Modal extra setup steps and swapping SANDBOX_MODE=docker for modal:

SANDBOX_MODE=modal decode --repo https://github.com/decodingai-magazine/building-a-coding-agent-from-scratch-course.git

After running it, Modal’s dashboard shows how many sandboxes are live (5 in our case):

And if you open the sandbox app and click “Sandboxes”, you get the full list:

These sandboxes are created when you enter a new Decode session and automatically cleaned up when you exit via the /quit command.

The wanted side effects of remote sandboxes

Remote sandboxes give you 2 powerful side effects beyond safety.

The first is compute. Modal sandboxes accept a GPU spec like gpu="B200:8" at creation, letting you agentically fine-tune models or process large datasets (eg extracting knowledge graphs from 1000+ documents) with open-weight LLMs such as Qwen3.6, Gemma 4, K3, or GLM5.2.

The second is scale. An orchestrator agent running directly on the host (no sandbox) can hand work to background agents running on Modal remote sandboxes. Put them on a CPU if you just need a bunch of parallel agents hitting an LLM API to pull tickets from your Linear backlog, or on a GPU if you need to squeeze out more juice.

In this case study, Ramp, a fintech company, runs every agent session in its own Modal sandbox with a full dev environment inside. Modal’s own conclusion from the case study is that with cheap isolation, the bottleneck shifts from “can the agent write correct code” to “how many agents can you run in parallel”.

Next steps

Should you sandbox all the time? No. As a Claude Code power user, to keep it simple, I still run directly on my machine in folders versioned by git or Obsidian Sync.

Sandboxes are non-negotiable for:

• 24/7 personal assistants that control your whole computer like OpenClaw or Hermes;

• non-engineers using tools like Claude Cowork;

• unmonitored remote jobs like Codex;

• or when chasing GPUs and parallel scale.

Is this the only way to add sandboxes to your harness? As we are just getting started, surely not. For another perspective, here is Abhishek Bhardwaj’s talk, explaining how OpenAI runs agent workloads in a cloud of microVM sandboxes instead of containers.

In the next lesson, we will explore the key context-engineering techniques that coding harnesses use: memory, compaction, skills, and LSP servers.

Here is the course roadmap, lesson by lesson (see all in GitHub):

1. Building a Coding Agent From Scratch

2. The Bare-Bones Coding Agent Loop

3. From a Raw Shell to a Sandboxed Coding Agent ← you are here

4. Context Engineering for Coding Agents

5. Subagents Are Context Engineering

6. Swarm of Remote Agents

7. Agent Evals 101

8. Turn Your Agent’s Failures Into a Regression Suite

Or watch the video version of the lesson:

But here is what I’m wondering:

Do you run your coding agent raw on your machine, or sandboxed?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

Special thanks to Modal, Opik (by Comet), and Kitaru (by ZenML) for sponsoring this open-source course and keeping it free!

Images & videos

If not otherwise stated, all images are created by the author.

I’m on vacation, so today I will keep it short.

Since February 2026, when I got my $200 Anthropic subscription, I maxed it out only once. It happened when Fable was released, and I started using it the way I used previous models such as Opus. After 4-5 queries through my content and software factories, I was running out of tokens.

Since then, I still use Fable, but only where it’s worth it. Never hitting my weekly token limit. At least not by surprise. I still have those moments where I am aware that if I press Enter, it will cost me tokens. But sometimes it’s worth it.

Anyway, I wanted to share with you 11 context/harness engineering tips that I use in my daily workflows and that work like a charm for me.

1. Progressive disclosure is the foundation everything else sits on

I use references in my AGENTS.md files, LLM wikis, and Obsidian to build hierarchies, so the model walks a table of contents and pulls a file only when something resolves to it. Anthropic frames it as a finite attention budget: every token you add competes with every other one.

OpenAI rebuilt its AGENTS.md into a roughly 100-line table of contents pointing into a linted docs tree, and shipped about a million lines of Codex-written code that way. Quote: “give Codex a map, not a 1,000-page instruction manual”.

2. Plan like crazy, then let a cheaper model execute

Token-wise, planning is cheaper than executing. I use the most powerful model available (currently Fable) to produce a plan, then hand it to a cheaper one. I never say “implement X”. Instead, I build an extremely detailed plan going through the system design, architecture, components, interfaces and data flow. When a smaller model executes that, it’s hard to make a mistake.

I work the plan through 4 dimensions (known knowns, known unknowns, unknown knowns, unknown unknowns) and let the model grill me with questions to surface blind spots.

3. For greenfield work, an LLM wiki is the context you point at

A wiki lets a plan reference 30 documents without loading 30 documents. When I start working on a new project, I always build one first: a light context layer built via files instead of a database. The plan cites the index that describes the documents, not the corpus, so it stays grounded at near-zero cost. I wrote a full piece on this.

4. Send a cheap subagent to read the whole thing

The subagent pays the reading cost. You only receive the summary. When I’m parsing data, I spin up Haiku or Sonnet subagents that hand back only an executive summary to Opus or Fable. A subagent can burn tens of thousands of tokens and return a distilled 1,000-2,000. I clustered my entire Decoding AI archive this way and my main agent’s context barely grew.

5. An ADR and a glossary per feature

For every new feature my planning step outputs a task list, an architecture decision record (ADR), and a glossary update. The ADR caches the why behind each decision; the glossary pins down what each term means. Both act as context for later features. OpenAI recommends the same artifact — “exec-plans checked into the repo” with progress and decision logs.

6. I don’t graph-index my codebase

I don’t build graphs to index my codebase. The code and the docs drift apart almost immediately, and then you burn tokens keeping an artifact in sync with the thing it describes. That violates clean code principles before it’s a token problem.

Note that keeping a glossary and a log of ADRs is different from building a graph out of your code. The glossary and ADRs are orthogonal to the codebase, tracking your domain conventions and design decisions over time. If you change the code, the ADR log doesn’t need to be changed. It just gets a new entry.

Worth being precise, because Cursor is a live counterexample: it diffs its embedding index against file hashes every few minutes and re-uploads only what changed. A strategy that adds a lot of complexity. Meanwhile, Claude Code and Codex just parse the codebase at runtime via glob and grep.

7. Your modules are already the graph

Instead of indexing, I think hard about how to modularize my code, how to shape the interfaces, and how data moves between components. That is your graph, and progressive disclosure parses it really well.

My sanity check: can I paste this module into another project and have it work?

8. Trim the payload you never see

Tools like Claude Code become a Swiss army knife that puts a ton of stuff into the system prompt that you never use. They fill it with references to all their features, such as push notifications, remote triggers, scheduling, etc. The logic is simple: for the harness to work with all those features, they need to be mentioned in the system prompt.

The solution is either to go with a minimalistic approach such as Pi, which ships a bare-bones coding agent you can extend via plugins, or to trim down your Claude Code/Codex harness via a settings.json like this:

Here is how my context window looks now. Hooked into my content & software factory on a 1M-token window, my payload is 21.4k tokens: 4.3k system prompt, 11.6k system tools, 2.7k across 17 skills, and 46 MCP tools costing literally zero because they load on demand.

9. Turn auto-memory off — under one condition

Auto-memory files, like MEMORY.md, reload into the system prompt every session.

I set Claude Code’s auto-memory to false.

Turning memory off is correct when your continuity already comes from committed artifacts you control (AGENTS.md, ADRs, glossaries, LLM wikis).

10. The Caveman plugin

I also use the Caveman plugin to cut token usage. LLMs tend to be overly verbose. This plugin transforms the messages from full sentences into “caveman talk.”

11. The principles that keep every AGENTS.md small

Again, to keep the LLM from being overly verbose and to make it support each instruction with examples, I keep these 3 rules in every AGENTS.md file:

• Always prefer removing instructions over adding more.

• Always use minimum words that still achieve goal — explanations, docs, code.

• New rule in memory (such as AGENTS.md), resources, skills or other files → always support with clear, concise explanation + good and bad examples. Good examples: “a 200-token chunk size”, “sub-100ms latency”. Bad examples: “a powerful architecture”, “a robust pipeline”.

But here is what I’m wondering:

What technique are you using that is not on this list?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

Images

If not otherwise stated, all images are created by the author.

In LangChain’s Terminal-Bench experiment, changing only the harness (with the same model) moved a coding agent from ~30th place into the top 5: the harness, not the model, is what makes a coding agent good.

In the open-source course Building a Coding Agent From Scratch, you’ll build that harness from scratch in Python: Decode, a complete coding agent that grows lesson by lesson from a bare agent loop into a swarm of remote agents running in parallel in the cloud.

Why? You’ll be able to engineer custom harnesses for your own AI products (the skill behind that leaderboard jump), and you’ll understand what Claude Code and Codex actually do under the hood, turning you into a power user.

Lessons:

1. Building a Coding Agent From Scratch

2. The Bare-Bones Coding Agent Loop ← you are here

3. From a Raw Shell to a Sandboxed Coding Agent

4. Context Engineering for Coding Agents

5. Subagents Are Context Engineering

6. Swarm of Remote Agents

7. Agent Evals 101

8. Turn Your Agent’s Failures Into a Regression Suite

Full open-source course

Special thanks to Modal, Opik (by Comet), and Kitaru (by ZenML) for sponsoring this open-source course and keeping it free!

Lesson 2: The Bare-Bones Coding Agent Loop

When I first tried to implement the code for this course I got greedy. I tried to one-shot the harness in Python: a TUI, an agent, and a handful of tools like bash, read, write, and edit. It worked, until I started adding skills, memory, and compaction. That’s when everything started to fall apart. No “fix this “ prompt could have saved it. That’s when I realized that building a harness for a coding agent is a lot more strategic than just: “Hey, look at OpenCode’s open-source code and build a replica for me.”

So, I took a step back, deleted everything and started researching how Claude Code, OpenCode, Aider and Pi work. Only after I pinned down the whole architecture, with its components, interfaces and data flow I started to build Decode. The coding agent I will teach you how to build in this series.

In this article, lesson 2 from the open-source course Building a Coding Agent From Scratch, we will learn how to build a bare-bones coding agent from scratch. At feature parity with the popular Pi agent harness.

We will build the coding agent loop. See how the loop decomposes a goal into a plan and executes it. Understand which tools are critical for a coding agent, and which ones are nice to have. Build a TUI. See why naively sending messages from the terminal to the agent can corrupt the loop. For the full architecture of Decode, what we will build throughout this series, we recommend reading Lesson 1.

On top of that, we will plug in 3 LLM providers (Modal, OpenRouter, and Gemini) and understand when serverless computing such as Modal beats token-based APIs. Also, to easily debug the agent, we will integrate trace monitoring via Opik.

Example of running Decode (our coding agent) powered by Qwen 3.6 35B:

Let’s start with an end-to-end walkthrough of the coding agent harness.

One Turn, End to End

Our expectations from Decode’s coding agent loop are similar to any other coding agent you’re already used to: Claude Code, Codex, Pi, OpenCode, etc. You open it in a repository or any other folder, give it a goal, and let it loop until it completes the goal.

In our .decode/skills/demo-6-article-kg demo, managed as a skill, we ask Decode to pull data from multiple sources, extract key entities and concepts and render a KG in HTML.

The agent kept looping until we got the graph below. Pretty cool, right?

This is a great example where the agent had to break a goal into multiple steps and execute each one independently until the goal is reached.

More concretely, this is the agent’s whole lifecycle: plan → explore → apply → execute → observe → next task, repeated until the goal is met.

Everything starts with you typing your goal into the TUI and pressing Enter. The request goes into the steering queue and then into a fresh turn, and the loop sends the prompt and tools to the model, a Qwen3.6 served on Modal. It kicks off planning. Via todo_write it records 4 tasks (fetch → extract → render → verify). Via web_fetch Decode fans out in one step and scrapes all the sources. The model reads the scraped output and extracts the KG triplets into a graph.json file via its write tool — the first call that can change your disk, so the run stops and asks. You type y. Ultimately it generates the HTML code that loads the data from the JSON files and beautifully renders it by calling the write tool to create the kg.html file. Finally, it calls todo_write to update the TODO list and mark the goal as done.

Based on our open-source GitHub, here are all the components of the agent loop that transform a simple Pydantic AI agent into the foundations of any coding agent you are using out there:

src/decode/
├── agent/          # loop.py (the agent loop), factory.py
├── harness/        # runner.py (single-flight turns), queue.py (steering/follow-up)
├── tools/          # registry.py, files.py (read/write/edit/glob/grep), bash.py, askuser.py
├── context/        # session_log.py (append-only JSONL)
├── observability/  # tracing.py (Opik via OTLP)
└── tui/            # app.py (prompt_toolkit input), render.py (Rich output)

In this course, we will mostly follow Claude Code/OpenCode’s design, where we have a larger set of tools, memory and agent features built into the harness. Mostly for educational reasons, to explore all the options out there. But there is also a beautiful, minimalist approach to this, based on the philosophy of Mario Zechner, creator of the Pi harness: strip the agent loop back to its bare bones, keeping only the essential logic, to avoid adding noise into the context window and keep everything observable and in control. If you want more features, you add them explicitly as plugins, instead of getting new “space ship” cool-but-useless features with every version. Thus, to understand both perspectives, we will keep drawing parallels to Pi’s architecture.

The Agent Loop

In Lesson 1, we said that the agent loop is the good old ReAct pattern. The model reasons, picks a tool, the harness executes it, and the observation feeds the next step — an agent is just an LLM using tools on environmental feedback in a loop.

ReAct in the abstract is reason → act → observe. Give it a set of tools it can operate on a codebase and voilà, it becomes a coding agent.

Plan: todo_write the task list before touching anything.
Explore: read, glob, grep the codebase.
Apply: write and edit, stopping for your verdict.
Execute: bash the tests, the linter.
Observe: a failure is the next observation, and the repair is another Apply → Execute pass on the same task. Tick it off, re-enter.

In Claude Code’s leaked source the core loop is ~150 lines. To emphasize the harness around the agent, we will use Pydantic AI for the “core agent” and focus on everything around it (from build_agent in src/decode/agent/factory.py):

def build_agent(*, model: str | None = None) -> Agent[AgentDeps, str | DeferredToolRequests]:
    built_model = _build_model(model=model)
    agent = Agent(
        built_model,
        deps_type=AgentDeps,                       # cwd, task store, event emitter — the harness state
        output_type=[str, DeferredToolRequests],   # a step ends in text — or pauses on approvals
        output_retries=3,                          # empty / thinking-only turns get another attempt
        tool_retries=5,                            # only resets on success — the plan-survival budget
    )
    register_tools(agent)                          # the coding tool set
    _register_instructions(agent)                  # ONE merged system prompt

    return agent

The Agent owns one step: one model request, the tool executions it triggers as “actions”, and the results it gets back as “observations”, ending in either text or DeferredToolRequests. The agent loop is operated by the AgentTurnHandler (src/decode/agent/loop.py) class chaining steps together through agent.iter.

AgentTurnHandler.__call__ is an async generator that streams tokens in real time via Python generators. It runs a while True loop with two boundaries: yield Boundary.MODEL_REQUEST and yield Boundary.WOULD_STOP. Via Boundary.MODEL_REQUEST, we can inject (steer) new messages into the loop before it finishes reaching its goals.

class AgentTurnHandler:

    ...                 # ONE instance per session; owns the turn + history

    async def __call__(self, ctx: TurnContext) -> AsyncGenerator[Boundary, list[str]]:
        while True:
            steering = yield Boundary.MODEL_REQUEST
            if pending_results is not None:
                self._append_steering(steering)
                output = await self._run_turn(ctx, deferred_results=pending_results)
                pending_results = None
            else:
                output = await self._run_turn(
                    ctx, prompt=self._compose_prompt(next_prompt, steering)
                )
            if isinstance(output, DeferredToolRequests):
                pending_results = await self._resolve_deferred(ctx, output)
                continue
            self._persist_turn()
            follow_ups = yield Boundary.WOULD_STOP
            if not follow_ups: return
            next_prompt = "\n".join(follow_ups)

The whole action happens inside the self._run_turn method, which makes the actual model call, and based on the results from the model, via the run object, it either emits answers from the model or executes a tool request.

In more detail, this is what happens:

1. A fresh step passes prompt, a resumed step passes deferred_tool_results, never both.

2. The async for streams each node to the TUI as an event, so you watch tokens arrive live.

3. The finally carries run.all_messages() forward, so a step that dies keeps everything it accomplished in memory.

class AgentTurnHandler:

    ...                         # ...same class as above

    async def _run_turn(self, ctx, *, prompt=None, deferred_results=None) -> str | DeferredToolRequests:
        async with self._agent.iter(
            prompt,
            deps=self._deps,
            message_history=self.message_history,
            deferred_tool_results=deferred_results,
        ) as run:
            try:
                async for node in run:
                    if Agent.is_model_request_node(node):
                        await self._stream_model_node(ctx, node, run)
                    elif Agent.is_call_tools_node(node):
                        await self._stream_tool_node(ctx, node, run)
            finally:
                self.message_history = run.all_messages()
                self._last_input_tokens = _leg_input_tokens(self.message_history)
        return run.result.output

💡 The loop has no max-steps knob, based on Pi’s principles: “the loop just loops until the agent says it’s done.” A cap is a guess about how many steps a task needs, and the model already signals completion by returning text instead of a tool call. The signal you should be careful about is the context window.

The first thing every step does is call a model, and picking that model should always be a one-line config choice.

The LLM Providers

It’s an architectural mistake to couple your design to a single provider. The loop shouldn’t know whether it talks to Gemini, OpenRouter, Modal, or whatever comes next.

All provider knowledge lives in _build_model() in src/decode/agent/factory.py. You pick the provider by setting the LLM_PROVIDER environment variable, which is picked by your Pydantic settings object from src/decode/config/settings.py.

def _build_model(*, model: str | None = None) -> Model:
    provider = settings.llm_provider                     # explicit LLM_PROVIDER env var
    if provider == "gemini":
        return GoogleModel(
            model or settings.gemini_model,
            provider=GoogleProvider(api_key=_provider_api_key("gemini"))
        )
    if provider == "openrouter":
        return OpenAIChatModel(
            model or settings.openrouter_model,
            provider=OpenRouterProvider(api_key=_provider_api_key("openrouter"))
        )
    if provider == "modal":
        client = AsyncOpenAI(
            base_url=f"{settings.modal_endpoint_url}/v1",
            api_key=token_secret,
            default_headers={"Modal-Key": token_id, "Modal-Secret": token_secret}
        )
        return OpenAIChatModel(
            model or settings.modal_endpoint_model,
            provider=OpenAIProvider(openai_client=client)
        )
        
    raise ValueError(f"unsupported llm_provider: {provider!r}")

So from all the options out there (we know there are tons), why these 3? And no OpenAI or Anthropic. Because between them they cover the three tiers of the build vs. buy decision: Gemini is proprietary (you buy the model), OpenRouter is open weights bought as a service (you buy the serving), and Modal is open weights you serve yourself.

We choose Modal as our default option for 3 reasons. You pay for GPU compute, not tokens: agentic bursts fit GPU-time pricing. Serverless open-source models supported by Modal that we deployed are Qwen3.6 35B, Kimi K3, and GLM-5.2 via Modal Endpoints. If you want to use a fine-tuned model, you can easily swap it in with 0 changes on the harness side. Because it’s serverless, it scales to zero: you pay only when the agent is working.

This combination makes it ideal for intermittent periods of extremely high compute demand. For example, if you want to process 1000 documents, instead of paying for each token, which is what’s typical of all the APIs out there, you just pay for the compute time. To make testing Decode extremely affordable, for most of our tests we used Qwen3.6 35B, which runs on a single H200 GPU at $4.54 / h.

But going back to our 1000 documents example, if these 1000 documents translate to 1000 * 30000 tokens, that's 30M input tokens, plus ~500 output tokens per document. If we wanted to process all these tokens with Sonnet, that would translate to: 30M / 1M x 3 (input) + 500k / 1M x 15 (output) ~= $97. And prompt caching doesn't save us here — every document is a different prefix, so there's nothing to reuse across calls. Considering we could process this data with Modal at ~3000 tokens/second batched, that's under 3 hours of GPU time, which would have cost only ~$13. Modal benchmarked the same tradeoff and found that giving up interactivity, from ~200ms to ~4s end-to-end, provided an x8 throughput increase on identical hardware. If we do this multiple times per day, it quickly adds up. This is just napkin math, but you get the idea.

Now, on the other side of the spectrum, let's assume Decode sits idle while it waits on a “y” confirmation the agent asked for. If that happens overnight, you can easily add 10 hours of idle time to your GPU billing, an extra ~$45.

Alongside the pay-per-token model, which quickly becomes inefficient at high volumes, another dimension to consider is serverless vs. reserved GPU time. When you reserve GPUs, you have to reserve your peak. If your peak needs 40 GPUs on Friday afternoon and 2 on Sunday night, you sign for 40 — and you keep paying for 40 all weekend, all month, all year. With serverless, the meter follows the demand curve, so the total cost is just the rate R_s times whatever you actually used at each moment. The reservation instead charges you rate R_r times the peak, times the entire length of the contract T.

In other words, if reserving is 5x cheaper than serverless, but your peak is 5x higher than your average, then you break even. And the further your peak climbs above that price advantage, the better off you are paying for serverless. For the full math, see this blog post.

OpenRouter is the most popular gateway to every hosted model: closed source (Claude, OpenAI, Gemini) or open-source (Kimi K3, Z GLM-5.2, Gemma 4, etc.) behind one key. Thanks to their huge aggregated pool of models, they also have some free defaults, such as qwen/qwen3-coder:free, which we used in this course. Having them integrated into your app is a powerful way to easily experiment with different models without worrying about the underlying infrastructure.

We picked Gemini mostly due to their generous free tier, to provide you a third option to run Decode without any cost. Their models are rarely the best, but relative to other API-based models, especially OpenAI & Anthropic, they’re cheap and good enough to get most jobs done.

We recommend starting with Modal Endpoints, powered by Qwen3.6 35B by default, that spins up in a couple of minutes a configurable SGLangEndpoint. SGLang is the right default here because it and vLLM land in roughly the same place on throughput, but SGLang boots in ~1 minute against vLLM’s ~5 for small models, which is everything when your endpoint scales to zero. Modal’s $30 in free credits each month covers ~6.5 hours of H200 time, enough to run the whole course at no cost. And while Qwen3.6 35B is small next to GLM-5.2 740B or even GPT OSS 120B, it’s well tested with Decode and works well enough for our examples.

After you’ve completed the basic setup on Modal, you can start a new endpoint just by running:

modal endpoint create --model Qwen/Qwen3.6-35B-A3B-FP8 --env main

While trying it out, to avoid the cold start problem, we recommend setting the autoscaling limits to a minimum of 1 container:

🧑‍💻 Find the full setup docs in our GitHub repo.

But between Modal and OpenRouter, you can easily swap between hundreds of models. You can try out whatever you’re interested in. The only requirement is that it supports tool calling.

The System Prompt

The entire base system prompt is one paragraph:

You are Decode, a terminal coding agent that helps a developer in their working
directory. You are concise and precise: answer directly, prefer running the work
over describing it, and never invent file contents or command output you have not
seen. When you do not have a tool for something yet, say so plainly rather than
pretending.

The team behind Pi started this minimalist mentality, where you keep your system prompts, tools, and features to a minimum, to avoid filling your context window with noise that confuses the model more than it helps. This also makes maintaining and understanding the prompts, for us as humans, a lot easier. In the past months, OpenAI and Anthropic followed, trimming their system prompts by up to 70%, as more and more functionality around coding gets baked directly into the model’s weights.

Still, this paragraph is only the baseline. Pydantic AI builds a schema from every tool’s signature, which is added to the system prompt. Thus, each tool comes with a cost, meaning that the more tools you use, the more context window you consume, and the model becomes more confused about which one to pick. This becomes even more problematic when we start adding memory and skills into the mix, as we will see in future lessons.

Whichever model answers, it can only touch your repo through the tools.

The Core Tools

In reality, the tools are 90% of why this is a coding agent and not any other kind of AI agent. The agent needs to read files, execute programs, make HTTP requests. Pi ships exactly read, write, edit, and bash — under 1,000 tokens of prompt plus definitions, still respectable — top 10 on Terminal-Bench 2.0.

read   path, offset?, limit?   → file contents; first 2,000 lines by default
write  path, content           → create or overwrite; makes parent dirs
edit   path, oldText, newText  → exact-match surgical replace (whitespace included)
bash   command, timeout?       → run it in the cwd; returns stdout + stderr

On top of these 4 core tools, we have more specialized I/O tools such as glob and grep. Then there’s todo_write, used to decompose the goal into a plan of TODOs, web_fetch to fetch web pages, and ask_user to interact with the user.

💡 Pi doesn’t have a todo_write tool because it considers that the best way to store your plan is directly in a Markdown file, not in an in-memory TODO list the agent has to keep track of — a PLAN.md on disk beats an in-memory list.

Every snippet below opens with ctx: RunContext[AgentDeps]. RunContext is Pydantic AI’s per-run context object: the framework injects it as every tool’s first parameter and hides it from the model’s tool schema. The model calls edit(path, old_string, new_string) and never knows ctx exists.

AgentDeps is in our harness. We declare it once as deps_type=AgentDeps in build_agent, pass it into agent.iter(deps=...), and Pydantic AI hands it to every tool call. RunContext[AgentDeps] reads as “this run’s context, whose .deps is an AgentDeps.” It is the harness state, arriving by injection (from src/decode/agent/deps.py):

@dataclass(slots=True)
class AgentDeps:                                   # every tool's ctx.deps — the harness, injected
    cwd: Path                                      # the jail: file tools + bash resolve paths here
    emit: EventSink                                # push an event → the TUI renders it
    gate: PermissionGate                           # the allow / ask / deny policy (Section 6)
    resolve_permission: PermissionResolver         # a gate "ask" → the human's verdict, awaited
    resolve_user_question: UserQuestionResolver    # ask_user's answer, on the SAME channel
    task_store: list[Task] = field(default_factory=list)      # what todo_write rewrites in place

emit, resolve_permission, and resolve_user_question are used for dependency injection, so tools reach the terminal without importing the TUI, and they make this code easily testable by swapping in a different AgentDeps.

To keep things short and sweet, I will skip how read and write work and go straight to edit (see them at src/decode/tools/files.py). edit is a find-and-replace with one rule: load the file’s text, find old_string inside it, dump new_string into that exact spot, write the file back. But the devil is in the details. To make this replace work, it first needs to normalize away the invisible formatting the model was never shown — a file’s line-ending convention, a leading byte-order mark — so a match never fails on characters the model had no way to guess. Then it looks for old_string, and only when that misses does it retry with whitespace collapsed on both sides. Either way, the match must be unique: 0 hits returns not found, 2+ returns ambiguous, N matches, and both end up with ModelRetry:

def edit(ctx: RunContext[AgentDeps], path: str, old_string: str, new_string: str) -> str:
    if needs_approval(ctx):
        raise ApprovalRequired
    target = _resolve_in_cwd(ctx.deps.cwd, path)
    raw = target.read_bytes().decode("utf-8")
    final = _apply_edit(raw, old_string, new_string)
    _atomic_write_bytes(target, final.encode())

    return f"Edited {path!r} (replaced 1 occurrence)."

Within todo_write, the model sends the full desired list on every call and the tool overwrites the whole list — which is exactly what stops the plan from drifting out of the context window. The overwrite is in place ([:] = tasks, not a rebind) so the loop and the TUI keep pointing at the same list object, and every item is validated by the Task Pydantic model on the way in. It then emits one TaskListUpdated event carrying pre-rendered checklist lines: the tool maps pending / in_progress / completed to [ ] / [~] / [x] itself, so the renderer never has to learn the status vocabulary. The store is in-memory and per-run — nothing about the task list outlives the process (from src/decode/tools/tasks.py):

def todo_write(ctx: RunContext[AgentDeps], tasks: list[Task]) -> str:
    if needs_approval(ctx):
        raise ApprovalRequired
    ctx.deps.task_store[:] = tasks
    ctx.deps.emit(events.TaskListUpdated(tasks=_checklist_lines(tasks)))

    return f"Updated task list ({len(tasks)} task(s))."

That emit line is the harness’s entire UI protocol. events is the decode.entities.events module of frozen dataclasses (TurnStarted, ToolCallStarted, ToolResult, TaskListUpdated, ...) joined into one union known by the renderer.

Event = (
    TurnStarted
    | TurnFinished
    | AssistantTextDelta
    | ThinkingDelta
    | ToolCallStarted
    | ToolResult
    | PermissionRequested
    | AskUserRequested
    | TaskListUpdated
    | ContextCompacted
    | ContextMicrocompacted
    | AgentError
)

ctx.deps.emit is a synchronous event-based sink, injected into the harness to communicate with outer layers, in this case the TUI. This is clean architecture 101. The tool describes what happened as a typed value, and the TUI decides how to draw it.

web_fetch makes an async httpx GET request. What comes back is filtered twice before the model sees it — non-text content types are refused outright, and the decoded body is hard-capped at 2 MB, so an enormous page can never eat your entire context window. HTML is then mapped to Markdown keeping mostly the body and other metadata such as the title. Every failure path — non-2xx, timeout, connection error, wrong content type, HTML nested too deep to parse — throws a ModelRetry error, so a bad fetch is an observation the model has to handle explicitly and not ignore (from src/decode/tools/web.py):

async def web_fetch(ctx: RunContext[AgentDeps], url: str) -> str:
    if needs_approval(ctx):
        raise ApprovalRequired
    _validate_url(url)
    response = await _get(url)
    body = _render_body(response, _decode_body(response))

    return f"HTTP {response.status_code} {response.url}\n\n{body}"

ask_user lets the agent request input from the user. It emits AskUserRequested so the TUI renders the question, then awaits ctx.deps.resolve_user_question. Two things can go wrong: the agent loop can run in headless mode, where there is no interactive user at all, or the question can be dismissed (turn aborted, REPL shutting down). Both become ModelRetry text telling the model plainly to proceed without an answer (from src/decode/tools/askuser.py):

async def ask_user(ctx: RunContext[AgentDeps], question: str) -> str:
    ctx.deps.emit(events.AskUserRequested(question=question))
    try:
        return await ctx.deps.resolve_user_question(question)
    except NoInteractiveUserError as exc:
        raise ModelRetry(str(exc)) from exc

bash is the most powerful. Via bash the agent can execute any shell command such as ls, cd, mkdir, rm, cp, mv, and echo. Plus, piping and everything else you can run in a terminal. Which means that via bash the agent can run Python, TypeScript, Rust, or any other language. Via bash it can manipulate all the CLIs you give it access to.

Via bash it gets that magical feeling of being able to interact with the operating system and run any command.

But this also makes it the most dangerous one. rm -rf ~/ is always only a few tokens away. That’s why running your agent in a sandbox environment becomes so important. In a future lesson, we will get into the executor and how to run commands in local Docker and remote Modal containers.

Bash plus code execution is a big step toward giving the model a computer (from src/decode/tools/bash.py):

async def bash(ctx: RunContext[AgentDeps], command: str, timeout: float | None = None) -> str:
    if needs_approval(ctx):
        raise ApprovalRequired
    timeout_s = _resolve_timeout(timeout)
    result = await _get_executor().run(command, cwd=ctx.deps.cwd, timeout_s=timeout_s)

    return _render(result, timeout_s=timeout_s)

As we’ve seen at the beginning, the lifecycle of the agent is based on running these tools in order: todo_write plans, read / glob / grep explore, write / edit apply, bash executes (bash‘s exit code is the observation that starts the next pass).

When running .decode/skills/demo-6-article-kg that asks the agent to scrape 3 web pages and build a knowledge graph, the agent creates a 4-step plan, keeps track of it via todo_write, fetches the data via web_fetch, extracts the KG triplets directly, then via write saves them to graph.json. Next, it writes the HTML and CSS code that displays the KG triplets into kg.html. Ultimately, via bash it checks that the HTML code renders correctly. If not, the model edits the code and retries.

Adding Tracing to Understand What Is Going On

We’ve seen above how the agent calls the todo_write tool to bake the plan into its state. The trace is logged via Opik. Let’s see how easy it is to plug it into your own code.

Debugging your agent via the terminal is terrible. Not only is reading the full traces hard, sometimes it’s impossible, as you just don’t have access to all the spans, plus other essential metadata such as configs, token counts, tool inputs and outputs — and the list goes on. That’s why you need to plug in an observability tool like Opik from day 0 to get visibility into your agent’s behavior.

As we use Pydantic AI, it has native support for tracing via Logfire, the observability tool made by Pydantic. Logfire, like most observability tools, supports OpenTelemetry (OTLP) out of the box, which is the standard wire protocol for tracing. Thus, hooking up Opik is as easy as defining the OTLPSpanExporter to point to Opik’s managed servers via the endpoint parameter, plus authentication headers. As Opik is open-source, this would work exactly the same if you were hosting it yourself (from src/decode/observability/tracing.py).

def init_tracing() -> bool:
    key = settings.opik_api_key.get_secret_value()
    if not key:
        return False

    exporter = OTLPSpanExporter(
        endpoint=f"{base}/v1/traces",
        headers={"Authorization": key, "Comet-Workspace": settings.opik_workspace,
                 "projectName": settings.opik_project_name})
    logfire.configure(send_to_logfire=False, console=False,
        additional_span_processors=[BatchSpanProcessor(CostAnnotatingExporter(exporter))])
    logfire.instrument_pydantic_ai()

    return True

You can find the full setup on how to get started with Opik in our GitHub repository. Opik is optional to get through the course, but note that it has a generous freemium plan with 25k spans/month free. You can run the whole course for free.

In the GIF below, you can see an end-to-end trace of how the agent executed the .decode/skills/demo-6-article-kg skill that builds the KG from 3 articles. It consists of 47 spans, covering 6 steps, 15 model calls, and 18 tool executions over ~6m.

With the following token usage (no cost per token as we ran it via Modal):

cache_read.input_tokens: 631424
completion_tokens: 19781
details.cache_read_tokens: 631424
details.reasoning_tokens: 2912
prompt_tokens: 655624
total_tokens: 675405

The TUI and the Queues

You can build a TUI in 2 ways. Full-screen takes over the terminal’s viewport as a grid of cells (Amp, OpenCode): total layout control, but you lose scrollback, search and have to implement a bunch of features from scratch. Append-to-scrollback writes like an ordinary CLI (Claude Code, Codex, Pi). Decode takes the second approach.

As Python packages, we use prompt_toolkit to read the input and Rich to render output in append-style. Below you can see the loop that takes events from the harness and renders them on the TUI (from src/decode/tui/app.py).

with patch_stdout(raw=True):
    while True:
        submitted = await session.prompt_async(_PROMPT)   # one pinned input line
        intent, text = _interpret(submitted)              # Enter / Alt+Enter / Esc
        if decisions.pending:                             # mid-turn question? answer it
            decisions.resolve(text)
            continue
        await runner.submit(text, intent)

But what happens when you type a new command mid-turn, while the agent is still running your previous one? Injecting it instantly would corrupt the current tool call. Thus, you need to buffer your input on arrival and inject it only at boundaries, such as after a tool finishes running or a whole turn ends.

The input can arrive in 3 modes.

• Steering (plain Enter): the Runner executes it within a model turn, between tool calls.

• Follow-up (Alt+Enter): the Runner holds it until the turn stops, then runs it as one more step.

• Cooperative abort (Esc): sets a flag to stop the agent turn at the next boundary. Next, it clears both queues. This avoids corrupting the history.

As seen within the InteractionQueues class, for steering and follow-ups we use two different in-memory queues (from src/decode/harness/queue.py).

@dataclass(slots=True)
class InteractionQueues:
    steering: asyncio.Queue[str] = field(default_factory=asyncio.Queue)
    follow_up: asyncio.Queue[str] = field(default_factory=asyncio.Queue)
    def drain_steering(self) -> list[str]:  return _drain(self.steering)
    def drain_follow_up(self) -> list[str]: return _drain(self.follow_up)
    def clear(self) -> None:                # on abort: drop everything pending
        _drain(self.steering)
        _drain(self.follow_up)

This loop belongs to the Runner, which is the facade between the TUI and the agent. Its job is to keep the agent alive as long as the TUI session is running. It also takes care of draining the queues at the right moments and sending them to the agent (from src/decode/harness/runner.py).

class Runner:

    ...                 # single-flight: one turn in flight, ever

    async def _run_turn(self, turn_id: int, prompt: str) -> None:
        agen = self._turn_handler(ctx)
        boundary = await agen.asend(None)
        while True:
            if self._abort:
                aborted = True
                break                                 # Esc: stop at the boundary, keep history
            if boundary is Boundary.MODEL_REQUEST:
                sent = self.queues.drain_steering()   # steering: before every model call
            elif boundary is Boundary.WOULD_STOP:
                sent = self.queues.drain_follow_up()  # follow-up: only when the turn would end
            try:
                boundary = await agen.asend(sent)     # resume the loop; get the next boundary
            except StopAsyncIteration:
                break

This is the approach implemented in Pi. Another option, implemented in OpenCode, is to keep track of the session in your database, which means you can use your database as the queue as well. But that also means that you need to complicate your solution with a database. Which raises the question, how do we keep track of sessions?

The Session Log

In the interactive mode the conversation is the working state. To keep track of sessions, we create a session log — an append-only JSONL file per session under .decode/sessions/. This simple approach kills the need for a database entirely. This is how Claude Code implements their session management.

.decode/sessions/
├── 20260620T132411Z_d710a5b9-28af-418b-9a15-936942a8db12.jsonl
├── 20260620T132419Z_058cddc8-1d0f-406d-a754-098703c07cb5.jsonl
├── 20260622T110408Z_7114fe38-d294-4c78-bbc3-c454be57082b.jsonl
├── 20260622T111609Z_4bbf140e-3495-4d69-9a88-c7014fd77742.jsonl
├── 20260625T152011Z_7594e9b0-6c62-421a-ad74-9918c4d0291e.jsonl
└── ...

Every completed turn appends one messages line to the log. An aborted or crashed turn persists what it finished (from src/decode/context/session_log.py).

def append_turn(self, new_messages: list[ModelMessage]) -> None:
    if not new_messages:
        return

    payload = json.loads(ModelMessagesTypeAdapter.dump_json(new_messages))
    entry = {"type": "messages", "messages": payload}
    with self.path.open("a", encoding="utf-8") as handle:
        handle.write(json.dumps(entry) + "\n")

Based on the knowledge-graph example, here is how the log entries look:

{"type": "session", "version": 1, "session_id": "48900d84…", "cwd": "…/building-a-coding-agent-from-scratch-course", "created_at": "2026-07-16T19:32:01Z"}
{"type": "messages", "messages": [
    {"role": "user", "content": "The hardcore one: turn three live articles into a knowledge graph…"},
    {"role": "tool", "name": "web_fetch", "content": "HTTP 200 …/keep-knowledge-graph-clean\n\nHow to Keep a Knowledge Graph Clean…"},
    {"role": "tool", "name": "bash", "content": "Exit code: 0.\n\nstdout:\nNode count: 28 (should be 20-35)\nEdge count: 29…"},
    {"role": "assistant", "content": "## Knowledge Graph Visualization — Complete…"}]}

To resume the state, you can run decode --resume <session_id> to kick off a new TUI from an existing session log.

Next Steps

In this lesson, we’ve implemented a bare-bones version of the coding agent, adding the right tools, a TUI, and a way to manage sessions. More or less, at this point, we are at feature parity with Pi.

But as a course on harness engineering, we want to push it further. Any agent needs a headless mode (along with running it from the terminal), essential context engineering features such as memory, compaction, skills and guardrails such as the permission layer and sandboxes. That’s essentially harness engineering. So far we’ve built the agent loop. Now the real harness engineering begins.

In the next lesson, we’ll build the sandbox layer locally via Docker and remotely via Modal.

Here is the course roadmap, lesson by lesson (see all in GitHub):

1. Building a Coding Agent From Scratch

2. The Bare-Bones Coding Agent Loop ← You are here

3. From a Raw Shell to a Sandboxed Coding Agent

4. Context Engineering for Coding Agents

5. Subagents Are Context Engineering

6. Swarm of Remote Agents

7. Agent Evals 101

8. Turn Your Agent’s Failures Into a Regression Suite

Or the video lecture supporting the first two lessons:

But here is what I’m wondering:

From your perspective, as we’ve been using Claude Code, Codex, etc. for general productivity stuff, what makes a general-purpose agent a coding agent?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

Special thanks to Modal, Opik (by Comet), and Kitaru (by ZenML) for sponsoring this open-source course and keeping it free!

Images & Videos

If not otherwise stated, all images are created by the author.

Every AI application that wraps an agent is a harness!

In LangChain’s Terminal-Bench experiment, changing only the harness (with the same model) moved a coding agent from ~30th place into the top 5: the harness, not the model, is what makes a coding agent good.

In the open-source course Building a Coding Agent From Scratch, you’ll build that harness from scratch in Python: Decode, a complete coding agent that grows lesson by lesson from a bare agent loop into a swarm of remote agents running in parallel in the cloud.

Why? You’ll be able to engineer custom harnesses for your own AI products (the skill behind that leaderboard jump), and you’ll understand what Claude Code and Codex actually do under the hood, turning you into a power user.

Lessons:

1. Building a Coding Agent From Scratch ← you are here

2. The Bare-Bones Coding Agent Loop

3. From a Raw Shell to a Sandboxed Coding Agent

4. Context Engineering for Coding Agents

5. Subagents Are Context Engineering

6. Swarm of Remote Agents

7. Agent Evals 101

8. Turn Your Agent’s Failures Into a Regression Suite

Full open-source course

Special thanks to Modal, Opik (by Comet), and Kitaru (by ZenML) for sponsoring this open-source course and keeping it free!

Lesson 1: Building a Coding Agent From Scratch

In a test run by LangChain on Terminal-Bench, changing only the harness (same model throughout) moved a coding agent from roughly 30th place into the top 5. The model isn’t what makes a coding agent good. The harness is. So that’s what we’ll build from scratch. A coding agent harness.

When I first started digging into coding agents, I couldn’t tell where the agent ended and the harness began. I wanted past that black box, so in the past months I’ve researched how the most popular coding agent tools work under the hood: Claude Code (via its leaked source), OpenCode, Pi, and Aider.

That harness is the only layer you can actually engineer. And that’s why I think building your own coding agent harness from scratch is the best way to understand both how to better use coding agents to become a power user and how to build your own custom harnesses for AI agents.

During my research, I’ve built my own replica: “Decode: The Coding Agent”. Now, I want to show you how to build your own.

I don’t mean a simple LLM with some tools in a loop, which is just the good old ReAct pattern. I mean building the whole coding harness from scratch. Build a coding agent once, and you’re equipped to build a custom agent for any use case.

The truth is that adding custom business logic to existing open-source harnesses such as Pi, DeepAgents by LangChain, or Pydantic AI Harness is not that hardcore. But to know what to add based on the internals of these harnesses is extremely important. Those are fundamentals. The secret sauce that still makes AI Engineers valuable.

In this 8-lesson series, you’ll build a complete coding agent called Decode from scratch in Python. It runs as a CLI for interactive use or as a remote headless agent that lets you run a swarm of agents in parallel on a cloud platform like Modal for automation, such as picking up tasks from the backlog or handling PR reviews.

You’ll learn how to implement all of that in 3 phases. First, we will build the agent loop using Pydantic AI, powered by open-source models on Modal, OpenRouter, or Gemini, leveraging their free credits.

Then we will build the harness around the loop, adding context engineering, sandboxing in Docker and Modal, memory, skills, compaction, subagent support, and more! All the good stuff known as harness engineering.

Finally, the boring stuff that makes you a senior AI engineer: adding observability and building a custom evaluation harness similar to Harbor (the framework used to compute Terminal-Bench and DeepSWE, two of the most popular agent benchmarks). During this, we will implement a custom benchmark (because evaluating agents on public benchmarks never aligns with your business goals), agent regression tests via Opik, and trace replay via Kitaru.

Before getting into the implementation, I want to kick off this lesson with the most fun part: the coding agent's system design.

Let’s trace the architecture end to end, following one request through every component.

The High-Level System Design

You pass a coding agent a request — “fix the bugs that make the unit tests fail” — and it loops: explore the codebase, run the tests, read the feedback, edit the code, repeat. The input is your request, while the repository is the state of the codebase. The output is tested changes done to the repo. Everything in between is the harness.

We are pretty familiar with this process. But how does it work?

At the center sits the headless harness: no interface of its own. It exposes a module that interfaces connect to. At the core, it runs the agent loop every harness shares: the LLM picks an action (a tool call — run the failing test suite), the tool executes and returns an observation (2 failing tests, expected vs actual), and the loop feeds it back. Everything reads from and writes to the context window (the state of the LLM).

To highlight that the harness is independent from the LLM, we wanted to implement multiple LLM providers. The loop is powered by an LLM provider module that to support open-source models we implemented it with Modal (for self-hosting) and OpenRouter (for APIs). Also, we added support for Gemini for their free tier. Modal is the most interesting as it allows us to either host standard open-source models or easily host our own fine-tuned models on top of their serverless infrastructure.

That cycle lets the agent correct itself: implement the code, run or compile it, read the error, fix it — or catch the mistake before anything runs, with a Language Server Protocol (LSP) server flagging broken syntax the moment an edit happens. The tighter these feedback loops, the faster the agent converges on working code. The LSP server (we used ty - made by Astral’s - the guys behind uv and ruff) is the cheapest way to get feedback on code changes.

Along with the LLM providers and LSP server, the harness contains 4 more essential modules: Memory, Skills, Sandbox, and Permissions. Essential for context engineering and security.

The context window of the LLM is a budget. The conversation accumulates in the window until it stops fitting or performance degrades. Remember that too much context confuses the model, known as context decay. Every observation the loop feeds back spends from the same finite budget. Hence, the harness needs to implement compaction techniques such as summarization, truncation, or just clearing the window.

How do we interact with the headless harness? It has two core interfaces: interactive mode, a terminal user interface (TUI) connected to a single live session, and remote mode, where an agent runtime (Modal) runs N headless harnesses in parallel as serverless functions.

On top of everything, we have the AI evals & observability layer, powered by Opik and Kitaru, that records every model call and tool call and turns a bad prompt tweak into a failing regression score before your users feel it in production or a signal used to improve the agent.

Here is how all the components fit together:

You type the request into the TUI. The input enters the steering queue through the priority gate and into the session’s context window. The loop sends the window (with the available tools) to the model through the LLM provider. The model answers with an action — grep for the failing assertion, then edit stats.py. Permissions check whether it can execute the tool, the tool executes inside the Sandbox, and the observation is added back into the context window. The loop repeats until the model stops calling tools. During the loop, events stream back to your terminal in real time via an async generator.

Next, let’s zoom into the agent loop, where the agent ends and the harness begins- the boundary that defines this entire series.

The Headless Harness & The Agent Loop

The agent itself is a Pydantic AI agent. A ~20 line definition that composes the model, the tools, the output type and iterates until the model stops calling them. The snippet below is the distilled version of Decode’s loop:

from dataclasses import dataclass
from pydantic_ai import Agent, DeferredToolRequests

@dataclass
class AgentDeps:                # the harness state injected into every tool call
    cwd: Path                   # the repo the tools operate on
    emit: EventSink             # streams events to whatever interface is attached
    gate: PermissionGate        # allow / ask / deny, per tool call
    resolve_permission: PermissionResolver  # asks the human when the gate says "ask"

agent = Agent(
    build_model(settings.llm_provider),       # gemini | openrouter | modal
    deps_type=AgentDeps,
    output_type=[str, DeferredToolRequests],   # a turn ends as a final answer (str),
)                                              # or as tool calls paused for approval
register_tools(agent)                          # read, edit, bash, grep, ...

async with agent.iter(prompt, message_history=history) as run:
    async for node in run:                     # model request → tool calls → repeat
        stream_events(node)                    # everything else is the harness

Through the AgentDeps dataclass we inject harness related components into the loop. The tools are never aware of the interface: TUI or headless. This is what I like to call “a practical clean architecture”, where you are obsessive about separating the serving and infrastructure layers from your core business logic via interfaces and composition.

output_type names the only 2 ways a turn can end: a final answer, or tool calls suspended mid-turn waiting for human input.

We will go into all the details about the agent loop, explaining how we hook multiple closed-source/open-source models, the tools and other deps into the loop, in the next article.

These ~20 lines are the entire tool-calling LLM agent. The thing people call “the agent” ends here. Everything we build on top of it across 8 lessons is the coding harness.

Pydantic AI is a thin layer over the model APIs: typed tool definitions, streaming, structured outputs, with no heavy framework between us and our own logic. What makes this a coding agent is the tools: read and edit files, run bash, search the codebase, manage a task list, fetch the web, ask the user a question, and spawn subagents. Plus, the LLM specializes in coding. Anthropic is well known for continually optimizing their models for coding tasks. That’s why with every release their models get worse and worse at writing, while better and better at coding.

Still, almost everyone already uses coding agents such as Claude Code or Codex for completely different use cases (including myself): productivity, content creation, finance, sales, marketing. The line between a coding agent and a general-purpose agent is blurry, as any agent needs to natively interact with your computer. How would a harness specialized in writing look? Other than making the LLM sound better. Which is not the harness!

On top of the agent loop, we have 6 modules + compaction that transform the agent loop into a headless harness.

The Six Modules + Compaction

LLM Providers — Decode ships with 3: Modal to serve any type of open-source models (pre-configured or your own fine-tuned version), deployed as Modal endpoints — serverless GPU web endpoints that put the model behind a URL the harness calls like any hosted API, OpenRouter to reach any hosted model behind one API, and Gemini for Google’s free tier for following along. Also, by implementing three providers we can showcase how to swap the model only with a config change, not a rewrite. The loop should never know which model is powering the harness. Clean architecture, remember?

Sandbox — an agent that runs bash can break things. We wrap the bash and file-I/O tools to execute inside a sandbox (Docker locally, Modal Sandboxes remotely) instead of on your machine.

Permissions — the permission layer is the most important guardrail: it asks you before running every action, or only the risky ones, depending on the mode. For example, it pauses before a destructive bash command like rm -rf and waits for your approval. This is similar to Claude Code’s default, edit, and auto modes.

Memory — the iconic AGENTS.md that carries your project instructions plus a MEMORY.md that encodes what the agent learns (e.g. “run the tests with pytest -x, never the full suite”). Plain files, deliberately no memory database and no codebase index: the repo is explored dynamically with grep. Just-in-time reads beat a stale heavy index.

Skills — the omnipresent skills feature that encodes reusable workflows loaded only when invoked, so the context window doesn’t get bloated with all your instructions at once. The repo already ships a set of skills under .decode/skills/ that you can try out of the box.

LSP Server — a key component of a coding agent. It keeps an index of all your variables and functions. The loop gets syntax and semantic information about the codebase through 2 channels instead of brute-forcing through it or waiting for the code to run. On demand, the model asks where a symbol is defined, who calls it, what its real contract is, and what’s broken in a file. The model gets 1 precise file:line answer. After each edit, a fast syntax checker feeds its findings back into the loop. The agent fixes a type error in the same turn, before running/compiling any code. The cheapest feedback loop in the system.

Compaction — not a module but a behavior of the harness itself that helps keep the context window as small as possible. The most iconic strategy is to compact the window once it goes over a threshold, squashing the window’s old head into a summary + keeping the recent interactions fresh: [summary, *tail]. You can also call this manually via /compact or simply clean the whole context window via /clear.

That’s the headless harness. Now, let’s plug it into the interfaces you use every day.

The Interactive Mode

The first is the TUI: the interactive mode, a terminal interface wired directly to one live session of the headless harness, in memory, in the same process.

We built it in Python, where prompt_toolkit handles input and Rich handles output of the terminal experience. It’s append-style, like a conversation log, rather than a full-screen app. Keystrokes go in, and the session’s event stream comes back in real-time via Python generators: streamed text, tool calls, approval prompts.

Why Python? To be honest, we picked it mostly for teaching purposes.

From my recent poll, I realized that all my audience knows Python. So this makes it accessible to you. But if I were to ship this for real, I’d reach for TypeScript (the language claude-code, OpenCode, and Pi chose) or Go, which compiles to one tidy binary. Still, Aider proves Python can carry a serious coding agent.

And in this series we will focus on the “why”, the design and architecture decisions. Thus, the programming language is not that important and you can easily swap it out for another language.

The steering queue

What happens when you send a new command and the agent is already mid-task?

Injecting it the moment it’s sent would corrupt a tool call already in flight. Ignoring it makes the agent unsteerable. Every serious harness hits this: Pi drains steering messages mid-turn but follow-ups only at the turn boundary, and claude-code ranks queued input so user messages never starve.

Decode’s answer is the steering queue + priority gate. Input is buffered the instant it arrives into the steering queue and injected only at a safe boundary: before the next model call, never mid-tool-call. The gate decides how to process the messages from the queue: steer now, answer later, or abort.

The steering queue can be a simple in-memory FIFO queue. No need for a complex queue infrastructure like RabbitMQ.

Try it out

You can already clone the Decode repository and try it out for yourself.

The repo has all the setup instructions. There are $0 costs to try it out. We recommend starting with Modal, which offers $30 in free credit every month.

The experience is similar to what you already know: you run decode in any repo and the TUI opens as a fresh session.

We bundled together a few demo’s as skills under .decode/skills/, so you can easily try out the coding agent with 0 friction:

Such as creating a snake game:

Fetching data from a GitHub repo and rendering a web page:

Or something more complex like scraping data from the internet, extracting a knowledge graph of entities and topics and rendering them into a cool visualization:

What if we want to run the agent remotely?

The Remote Mode

In the remote mode, we run the headless harness on Modal serverless functions. Now you can easily run Decode in the background without depending on your laptop and fully automate your agentic infrastructure.

You can trigger it from your CI/CD pipeline or Slack messages. A standard use case is to use cron jobs to pull tickets from Linear or Notion, launch one agent per ticket in parallel, and get notified when the implementation is done with a link to the ticket's PR.

Or even launch 3-5 implementations of the same ticket in parallel and let a frontier foundation model judge the candidates and rank them for you.

As in the image above, we will deploy decode as serverless functions on Modal, triggered via the CLI, webhooks, or cron jobs. The pros of this design are that you pay only for what you use, avoiding the need to run a server 24/7. Plus, because the LLM runs on a different inference endpoint and the tools execute within the sandbox, we can keep the application server extremely light.

There are two core infrastructure components:

• The execution layer, where the headless harness runs, is hosted on Modal.

• The sandbox layer, where filesystem tools (read, write, edit, bash) execute, is deployed as Modal remote sandboxes. This is optional. If not configured, the filesystem tools run on the execution layer.

If a tool isn't flagged to run within the sandbox, it executes directly within the harness, which is fine for planning tools.

Try it out

You can run the remote version by going to our course repo and setting up Modal, following our Modal and sandboxing tutorials (takes ~5-10 mins). Then everything is as easy as running:

uv run decode remote deploy
uv run decode remote run "<task>" --repo <url> --sandbox-mode modal

One question decides whether this engineering is worth it: “Can you prove the agent works for my custom use case?”

The Observability and AI Evals Layer

The evals layer answers 3 questions, each with its own mechanism.

Internal benchmarks answer: “Does it work?” These are the metrics you try to beat with every new feature. You can use public benchmarks such as Terminal-Bench to compare your agent’s performance against others. But the real deal is to build a custom benchmark suite from your own tasks that optimizes for your specific use case.

Each case is a task instruction, a fresh sandboxed environment, and a hidden oracle test that decides pass or fail. One example case is “add a --json flag to decode run“, with a hidden pytest asserting the output parses. It’s the same pattern SWE-bench applies to real GitHub issues and Terminal-Bench applies to terminal tasks.

Your own tasks predict your agent’s usefulness better than any leaderboard.

Regression tests answer: “Does it still work?” Every new feature re-runs a regression suite and compares scores against the baseline, so a prompt tweak or a new tool can’t silently make the agent worse.

Production evals: “Does it keep working?” Every session is traced with Opik, Comet’s open-source LLM observability platform: each model call, tool call, and token count is a trace, while the conversation is logged as a thread. Live scoring tracks quality in production, where you run a bunch of AI eval metrics on sampled traces to detect unwanted behavior and create warnings or alarms.

To implement all three cases, plus to monitor and store all our traces, we will use Opik as our evaluation harness and observability platform.

AI Evals on Steroids

With tools like Opik, you can only evaluate existing traces, but you often want to answer questions like: “What would have happened if I used a different model or reasoning level?”

That’s why a popular strategy gaining traction is replays, via tools like Kitaru. You take a recorded trace, change one thing (the model, the system prompt, a parameter), and re-run the whole session from the top, then compare it against the unchanged run as your baseline.

The tools don’t execute again by default: when the replayed model makes a tool call that matches one from the original session, it gets the recorded output. When it calls something new, the policy you pick decides what happens: return an error so the model has to adapt, or run the tool for real. This way, you can isolate the impact of a single change on the model’s decisions, and replays cost no sandbox time.

Applying this strategy can drastically speed up exploration when optimizing your coding agent, whether you're trying to squeeze better scores on your benchmarks or find edge cases in your regression tests. You can also run error analysis on your results to detect and fix the most frequent and/or important errors.

Kitaru works in parallel with Opik. You can import all your traces from Opik or plug in a Kitaru adapter on top of your agent loop to collect them.

Next Steps

You’ve seen a bird’s-eye view of what a coding agent looks like: one headless core, a small tool-calling agent inside a much larger harness, 2 modes, and an evals layer that measures performance.

In the next lesson, we get into the code and start building the agent loop hooked into the TUI.

Here is the course roadmap, lesson by lesson (see all in GitHub):

1. Building a Coding Agent From Scratch ← you are here

2. The Bare-Bones Coding Agent Loop

3. From a Raw Shell to a Sandboxed Coding Agent

4. Context Engineering for Coding Agents

5. Subagents Are Context Engineering

6. Swarm of Remote Agents

7. Agent Evals 101

8. Turn Your Agent’s Failures Into a Regression Suite

Or watch the video lecture that supports the first two lessons:

But here is what I’m wondering:

Which part of your daily coding agent is still a black box to you: the loop, the sandbox, the memory, or something else?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

Special thanks to Modal, Opik (by Comet), and Kitaru (by ZenML) for sponsoring this open-source course and keeping it free!

Images

If not otherwise stated, all images are created by the author.

Agent memory is one of the hottest problems in AI engineering right now, with Graphiti, mem0, HydraDB, and cognee all racing to solve it. A race this crowded means one thing: nobody has cracked it yet.

But if you’re thinking of jumping straight into an agent memory tool without understanding how it works under the hood, read this first.

Here is what happened in one of my tests: LangChain’s MongoDBGraphStore gave me a knowledge graph (KG) in 10 minutes, and from 5 documents it invented 17 node types and 34 relationship types, where part_of, Part Of, and part of were three different ways to express the same relationship.

That’s why it’s incredibly important to understand how the memory layer works under the hood, so you know its limitations, how to properly configure it for your data, and how to plug it into your agent.

Or even decide if you need a unified memory at all, and if so, which tool to use.

Claude Code comes with a built-in memory layer powered by files. Is this enough for you or not?

Then, if you move to a more powerful solution, should you use a vector database, a graph database, or a combination of both? Do you need temporality or versioning?

And ultimately, how should you serve your memory layer to your agent? Through an MCP server, a CLI, or plain skills?

The reality is that there is no blueprint on how to build your own unified memory layer. Too many choices, too many trade-offs.

But that’s why I want to show you how to build a unified agent memory from scratch, in the most complicated way possible: via knowledge graphs (KGs). Plus all the other moving pieces: ingestion, querying, and serving.

Why? Because all the popular vendors (cognee, Graphiti, Neo4j’s agent memory, etc.) go in this direction. Plus, if you know how this works, it will be incredibly easy for you to build simpler solutions.

So, I am not saying that KGs are always the way to go. They are not. But they are definitely something you need to understand. At least to know when to stay away from them.

Let’s go!

A Session With a Unified Memory

First, here is what I expect a unified memory plugged into a harness to do.

I open Claude Code and ask what I need to do today. The memory already knows: building a coding agent for my next open source project.

I ask it to gather everything relevant I have on building a coding agent: Obsidian notes, Readwise highlights, starred repos, all ingested long ago. Then to build an LLM wiki on top of that as my agent memory while I implement the plan.

When I’m done, the conversation is ingested too, so the next session already knows all my decisions around the project.

Nothing in that session lived in the harness. Claude Code was just the interface.

So how do I implement a unified memory that supports this? Let’s kick off with the architecture.

The Architecture, End to End

Sources in, subgraphs out. In between sit 4 pieces: an ontology, a write path, one collection, and a serving layer.

The whole database layer is powered by MongoDB: a document warehouse plus a knowledge-graph store doing text, vector, and graph search in one collection. Because we use a single database, lineage is free: nodes hold references to their source documents, not copies from a separate source. This works incredibly well at 2-3 hops at query time.

OG databases such as MongoDB scale really well up to millions of documents via sharding and replicas.

Writes are 2 durable pipelines. A data pipeline normalizes each source into the warehouse. A memory pipeline cleans, chunks, extracts, normalizes, embeds, and writes the knowledge-graph objects. Both pipelines are powered by an orchestrator, such as Prefect or DBOS, to serve them, scale them, and make them durable.

On the read side, we offer multiple ways to interact with the unified memory: standard graph search encoded directly into the tool, agentic search where the agent writes its own MongoDB query, and a deep-search primitive that creates an on-demand LLM wiki for exploring larger subgraphs via progressive disclosure. Reads never touch the orchestrator, as they need to be fast and don’t benefit from the orchestrator’s durability or scheduling features.

The memory is reachable only through a FastMCP server exposing agent-shaped primitives (query, deep-search, ingest), never raw database operations. The MCP server should never be an API wrapper, but directly expose business logic to the harness. A hook fires the conversation ingestion as the session runs, so what the agent learns flows back into the memory — a process known as “continual learning.”

The most interesting part is not the database, the ingestion, or the query logic, but the ontology that makes everything possible.

The Ontology Is the Contract

An ontology is a contract between the writer and the reader. The LLM extracts against it. The query layer reads against it. Define it once as Pydantic, serialize with model_json_schema(), and dump that JSON into both writing and reading system prompts. 1 artifact, 2 consumers, no drift.

The ontology defines what the knowledge graphs look like. All the KG instances are extracted by passing the input data and the ontology schema to an LLM.

The memory is 3 layers. The long-term layer is the knowledge graph: the POLE+O entities plus preference and fact. A short-term layer holds the dialogue as conversation and session nodes. A reasoning layer records the agent’s work as agent, tool_call, and memory nodes.

One powerful ontology design is based on the 5 POLE+O nouns from law-enforcement analysis: Person, Object, Location, Event, Organization, which Neo4j Labs writes up as the POLE+O data model. Each noun carries an optional subtype used to further refine the ontology within a specific domain. For gaming, this can look like:

• Person: Player, Character

• Object: Item, Quest

• Event: Raid, Battle

Because the agent memory is often used for a personal assistant, you also need to plug preference and fact nodes into the ontology. A fact is an atomic subject-predicate-object triplet, an island no edge may touch, reachable only by similarity. A preference is the personalization layer with typed slots.

nodes: person · organization · location · event · object
       preference · fact
edges: related_to, with semantic_type ∈ {
         knows, member_of, employed_by,
         owns, uses, located_at, resides_at,
         alias_of, has_task, ... 
       }

Now, on top of the KG instances extracted by the LLM, we can also have structured nodes that are directly inferred via code.

Such as:

nodes: document · chunk
edges: part_of · next · mentions · referenced · has · same_as · superseded_by

There is a lot more to talk about ontologies. I have a full article just on this.

With the contract fixed, let’s see how we can ingest KG objects into the unified memory based on the ontology schema.

The Write Path

There are 7 stages in one direction. Chunk the document (512-token chunks, 64 overlap). Extract nodes and edges with an LLM. Validate. Resolve names. Embed. Deduplicate. Upsert into the MongoDB unified memory.

During the LLM extraction, we pass in a chunk and we receive a JSON response with nodes and edges, the same shape Neo4j Labs lands on in How Entity Extraction Works.

The model sees one chunk and nothing else. It receives no IDs and no prior state. It returns JSON. Every edge endpoint points to a node’s name in the same record. This allows us to scale the extraction pipeline through batching to process large documents of 1,000,000+ units.

Validation is done via Pydantic. We load the JSON response into a Pydantic model and validate it against the schema.

Name resolution only checks whether two canonical names match. It’s done purely on the name and its aliases: alias, exact, fuzzy 0.85, semantic 0.80, filtered by type.

But there is a next step, where we actually check if an entity is the same instance as another that already exists in the graph.

Deduplication runs on embeddings computed on the whole node content: ≥0.95 cosine merges, ≤0.85 makes a new node, and the gap writes a same_as edge for a human to judge.

So why two steps?

Name resolution simply helps us group entities by their canonical name. For example, “Demis Hassabis” and “Demis Hassabis, CEO” would both be resolved to “Demis Hassabis”. Or “Apple” and “Apple Inc.” would both be resolved to “Apple”. Super powerful for analytics and human curation.

Deduplication is the sensitive one — it decides what actually merges. It shouldn’t flag “Apple” and “Apple Inc.” as duplicates, as they are different entities with different meanings.

A wrong merge is the only unrecoverable mistake. That’s why, when the similarity score has low confidence, between 0.85 and 0.95, we don’t merge the nodes automatically, as the operation can be irreversible.

I break the full algorithm down in Keep Your Knowledge Graph Clean.

Which leaves the question that decides your RAM bill: once the triplets are extracted, how do we store them in the unified memory?

Append-Only Logs vs. a Single Knowledge Graph Collection

There are 3 designs, and the winner? Well... It depends... As always.

An append-only log plus a materialized view buys versioning and soft-delete for free, but you pay for it. Nested relationships duplicate edges and make them awkward to query. Separate edge docs give queryable edges, no duplication, and the cheapest footprint, at the cost of built-in temporal history and versioning.

Because the graph is a projection of the append-only log, a bad extraction is fixed by invalidating its event. You can change the materialization logic and replay it without re-extracting a single source. Also, full-graph temporality belongs to the log alone.

The log’s real cost: a vector index is at least as large as the data it covers. Indexing both the log and the view inflates ~10 GB of data toward ~40 GB of RAM. Index only the view, leave the log cold on disk, and it collapses back toward ~10 GB. But the idea is that adding an append-only log brings a lot of engineering effort you then have to maintain.

So think three times before you sign up for it.

At query time, the agents see the materialized view, not the log itself. Given the RAM cost, reach for the log only when versioning or the time dimension is core to your business logic. Otherwise, keep it simple and stick to a single knowledge graph collection.

Within the data model, Node IDs are {user_id}:{type}:{name}. Edge IDs are {source}|{type}|{target}. Because the key is content-derived, every write is idempotent.

The reality? You can get temporality even without a dedicated append-only log. Preferences and facts carry valid_from / valid_until, and a superseding preference writes a superseded_by edge.

Three Ways to Read the Memory

The 3 retrieval methods below are the serving layer’s search primitives, the tools the MCP server exposes to the agent.

Graph search is the default retrieval, baked into a single function. It runs $vectorSearch and $text in parallel over the KG collection. Fuse the ranks with Reciprocal Rank Fusion (RRF). Keep the fused top 10 as seed nodes, then expand with a bidirectional $graphLookup at 2 hops by default. That multi-hop step is the entire difference between RAG and GraphRAG.

Deep search widens the walk to 3 hops and persists the result to disk as a memory wiki the agent can access via progressive disclosure. The LLM wiki is a mini-graph of its own: you keep the (entity, relationship, entity) triplets by cross-referencing files, where each file is an entity. You can see it as a local cache that is either discarded between sessions or kept in sync with the KG collection.

NL query is agentic search that lets the LLM write its own MongoDB queries based on the ontology. Here you have to validate that the query is syntactically correct, and especially guard it against harmful intents such as deleting or modifying data. It’s safer to use this mode purely for read operations.

RRF replaces a reranker (a second model that re-scores the retriever’s top candidates) you don’t need yet. It is 20 lines of Python, needs no model, and runs in microseconds. Reach for a cross-encoder (a heavier model that compares the query against each candidate) when you search millions of heterogeneous documents.

When do you need to switch from a single database (MongoDB) to a graph database (Neo4j)? Past 4 hops, past ~100M to 1B vectors, when graphs are your business logic, or purely to visualize them: that is where you leave MongoDB for Neo4j. It’s common to use a single database in production (MongoDB) and a graph database as an internal tool for data exploration and analysis (Neo4j). This is a powerful combo, because for an internal tool you don’t care that much about 99.99% SLAs, while you can fully leverage Neo4j’s Cypher query language to explore and analyze your graph data.

Open, Closed, and Small

Extraction (from ingestion) and query translation (from agentic search) use gemini-3.1-flash-lite. Embeddings are Voyage voyage-3.5 at 1024 dimensions. Both are closed APIs.

You can take this further and fine-tune an open-source SLM (such as the Liquid family of models) as an extractor, query translator, or embedding model on your own data.

The issue with SLMs is that without fine-tuning they perform poorly on new data. But the beautiful part about them is that after fine-tuning they perform as well as a frontier model on a scoped task, while being much cheaper and faster to run.

I won’t go too much into fine-tuning here, but the idea is that at first you always want to start with an API that quickly gets stuff done, and then slowly replace the most important parts with your own fine-tuned models. You usually have to decide this based on cost, latency, performance, data privacy, or other dimensions important to your use case.

Build vs Buy: When Not To Build This

Everything above assumed you build everything yourself. But in reality, there are 3 levels to choose from.

Level 1: build the memory, the logic, and the serving layer yourself on one database.

Level 2: keep only the business logic and serving layer on top of a memory SDK, such as Graphiti, neo4j-labs/agent-memory, or mem0.

Level 3: run an off-the-shelf engine, such as cognee, managed Zep, or HydraDB.

This is a problem about owning your context layer, which I have a full article on.

Buy when you don’t care that much about customization. Build when you want your own solution. But since building everything yourself usually isn’t worth it (or you just don’t have the time to do it — even with Claude Code), the best solution is Level 2, where you own the business logic and can customize the serving layer to your needs.

But honestly, because things move so fast and I am lazy, my long-term memory still lives in Obsidian, Readwise, and Google Drive, plus a serving layer that creates scoped LLM wikis per project as agent memory. This way, you create mini-graphs for one specific problem, powered purely by .md files, without any infrastructure hassle. The reality is that this works for personal use only, and not so much for shipping a production-ready solution. I write about that approach in LLM Wikis as Agent Memory.

Still, I am applying the strategy based on KGs and ontologies to build Pulse, a data mining tool that runs the exact same strategy presented above on a knowledge-work ontology to extract the top 0.1% signal from the AI industry. I want it custom because I want it to be super light, powered by LadybugDB, and to leverage my Claude subscription for the extraction and deduplication part, built as a hybrid between skills and Python modules.

But here is what I’m wondering:

If you keep dumping everything into the unified memory, where do you draw the line between what your agent should remember forever and what it should be allowed to forget?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

Special thanks to MongoDB for sponsoring this article and keeping it free!

Explore Next

1. Iusztin, P. (n.d.). Own Your Context Layer: Portable AI Agent Memory. Decoding AI. https://www.decodingai.com/p/the-context-layer

2. Iusztin, P. (n.d.). Ship a Knowledge Graph Ontology in 5 Minutes. Decoding AI. https://www.decodingai.com/p/ship-a-knowledge-graph-ontology-in-5-minutes

3. Iusztin, P. (n.d.). How to Keep Your AI Agent’s Knowledge Graph Clean. Decoding AI. https://www.decodingai.com/p/keep-knowledge-graph-clean

4. Iusztin, P. (n.d.). LLM Wikis as Living Memory for AI Agents. Decoding AI. https://www.decodingai.com/p/llm-wiki-agent-memory

5. Neo4j Labs. (n.d.). Why Neo4j? Graph-Native Memory Architecture. Neo4j Agent Memory. https://neo4j.com/labs/agent-memory/explanation/graph-architecture/

6. Neo4j Labs. (n.d.). POLE+O Data Model. Neo4j Agent Memory. https://neo4j.com/labs/agent-memory/explanation/poleo-model/

7. Neo4j Labs. (n.d.). How Entity Extraction Works. Neo4j Agent Memory. https://neo4j.com/labs/agent-memory/explanation/extraction-pipeline/

Images

If not otherwise stated, all images are created by the author.

I spent 18 months turning my second brain into my living research memory. My digital life contains 10,994 notes: over 5,000 in Obsidian, another 5,000 in Readwise, plus more in Notion and Google Drive, growing by roughly 250 a month.

The reality is that most of my notes and bookmarks transformed into a graveyard. When I start a new article or open a codebase, I can’t recall the high-signal notes I already have, so I work without them or burn an hour digging.

The reflex is to reach for Codex or NotebookLM. But those are harnesses: they sit on top of your work, and the moment the chat ends, the context window loses all of it.

More context isn’t the fix. You re-paste the same links and re-explain the same project every session, while past research sits inert in a vault your agents can’t reach.

So , CTO of Towards AI, and I built the alternative, and we’ll show you how: your own AI Research OS. It’s a memory layer that sits between your second brain and any harness (Codex, Claude Code, or your own). It runs deep research across your notes and the open web, and stores what it finds as an LLM wiki you can query, maintain, and grow.

We first touched on this subject in a talk at the AI Engineer World’s Fair. After we saw that people loved it (it was published as a keynote in the online tracks - out of 82 videos - and is the 2nd most popular one), I decided to write an article about it as well.

By the end you’ll learn how to turn your 10,994 notes into a queryable LLM wiki that your AI agents can use as their agent memory, useful for research, coding and content creation.

You will build an LLM wiki as in the image below, where the payoff already compounds with just 12 sources:

Or go crazier with a larger wiki:

By the end, you will also see why the wikis don’t contain all my 10,994 notes to keep ingestion costs low. Let’s go.

Why a Bigger Context Window Won’t Save You

Most research doesn’t need this system. If you want a quick answer or a one-off, Google it, or reach for Codex and Claude Code. You need this only when the research has to stick and an agent has to reuse those sources later.

You don’t need a giant vault either. This is about topic density, not size. It already pays off on a topic-sized cluster: around 20 notes on one subject, or 2 or 3 repos, like coding agents such as Pi, OpenCode, and Aider. Louis-François runs it on a few hundred notes, not my 10,000+, and it still earns its keep.

Why not NotebookLM? It’s great at digesting sources, but you don’t own it, you can’t personalize it deeply, it isn’t agent-native, and it’s weak for coding since it’s browser-bound.

Why not a vector-database RAG pipeline? That’s the right call at production scale, but it’s infrastructure. It’s hard to inspect or edit by hand, and overkill for a personal tool you open every day.

So you build it yourself: a personalized research assistant that uses an LLM wiki knowledge base that’s light and human-friendly as its agent memory. The perfect context engineering technique as an interface between the human and the AI agent.

Now, I want to go over, step by step, how we reached our final deep research + LLM wiki design, with the “why” and “how” in mind.

The Deep-Research Loop, Version 1: Mining the Public Web

A year ago, building the Agent Engineering course, we scoped the loop tightly: give it a topic plus a few hand-picked golden links, and get back a single static research.md.

The deep research algorithm first scrapes the golden links for seed context. Knowing them up front lets the agent frame better questions. Then it runs query rounds: one orchestrator generates the questions, and a sub-agent per question searches (Gemini grounded in Google) and returns links plus a summary, which the orchestrator aggregates so the context never explodes.

Three rounds of 6 queries surface 40–50 links, which is too noisy to keep whole. A ranking step scores each source against the topic. Only the top-K get fully scraped, the rest are kept as summaries, and everything is compiled into one flat research.md. That version generated 35 course lessons fast.

It worked for the course, but it was aimed at the generic public web, with every golden link hand-picked from our own second brain.

Version 2: Point the Loop at Your Second Brain

Same loop, new target: aim it at your own sources instead of the public web, where you’ve already filtered what matters. Your second brain is a curated set of golden links, so the input shrinks to just a topic and the loop finds the rest.

You plug in Obsidian, Readwise, NotebookLM, and GitHub Stars, then extend with Gemini Deep Research, YouTube, Google Drive, or Notion. Code is a special case: a GitHub sub-agent clones a repo and builds a high-level architecture note.

The token-efficiency trick lives in the reranker. It scores each candidate from 0 to 1 against your question using only its metadata and summary, never the full text, then passes only the top-K to the model.

The new problem is that the output is frozen. You still end up with a static research.md, and real research isn’t static. You want another question answered, or part of it goes stale, and re-running the loop from scratch is expensive in tokens and time. The fix is a new layer that sits on top of the raw data: the LLM wiki.

Version 3: From a Static Pile to a Living Wiki

The pivot came from Andrej Karpathy’s idea of LLM-maintained knowledge-base wikis [1]. Instead of re-deriving knowledge from raw documents on every query, which is the RAG pattern, the LLM incrementally builds and maintains a persistent, interlinked wiki between you and your sources.

Since I realized the power of LLM wikis, I started to use them for all my personal setups as my agent memory. It’s simple, powerful and beautiful.

Since giving this talk, we found that Google had published its Open Knowledge Format (OKF), an open spec that “formalizes the LLM-wiki pattern into a portable, interoperable format” [2]. It’s built on the very blocks you’ll see below: plain markdown with YAML frontmatter, an index, a log, and links that form a graph. When Google independently ships the same architecture as a standard, you know the direction is right.

During ingestion it generates, as a byproduct, derivatives over your raw sources: entities and concepts, comparisons between them, notes tied to your questions, open questions, and a synthesis:

Storing each finding as an individual raw file, instead of one flat research.md, is the one change that makes everything downstream queryable and growable. Sources can include Obsidian, GitHub, Google Drive, even custom URLs (plain curl for simple sites, Bright Data for bot-walled ones).

The natural fear is that “index + wiki + query” means vector databases and knowledge graphs. It doesn’t, and that’s the surprising part.

A Memory Layer Built From Plain Files and No Database

Vector databases, knowledge graphs, and semantic and text search all add real complexity, too much for a personal research OS. So drop that infrastructure and build the whole thing on files and references: no database, just a simple index rooted in how your filesystem already works.

The index is the retrieval layer. An agent reads a single index.yaml first, a catalog with a summary and metadata for each source (original file, origin, title, authors, date).

There are 3 layers. A raw folder holds the immutable source data, a wiki folder holds the LLM-generated derivatives, and the index points to all of it. A real example is an index.yaml cataloging 10 sources and 38 wiki pages.

You even get a knowledge graph for free. Because everything is Obsidian-flavored markdown with references, Obsidian’s local graph renders the connections out of the box: entities like OpenCode or MCP linked to concepts like tool registry, context compaction, and sandboxing, and the sources they touch.

Obsidian is only a viewer, though. The whole system runs from any working directory through Codex or Claude Code, so the graph is a bonus on top.

Querying the Wiki, and Why It Never Freezes

The agent queries through progressive disclosure. It starts at the index.yaml summaries. If it needs more, it opens the source wiki page (an expanded summary), and often stops there.

If not, it follows references into the derivatives (entities, concepts, comparisons, notes). Only as a last resort does it read the raw source. Summaries are computed once, at ingestion, so the context window stays small.

The wiki is also alive. Ask a question and the LLM can spawn a new concept, note, or comparison, and each question is written to a log. So the wiki evolves as you talk to it, not only when you ingest data.

It’s never frozen, and it grows 3 ways: ingest a new custom link, run another deep-research round, or let it create derivatives purely from your questions.

One design choice makes this safe to run against a decade of notes: the wiki never sits directly on your whole second brain.

Scope It to a Project With PARA

Your second brain stays an immutable snapshot. I organize everything with the PARA method: Projects, Areas, Resources, Archive [3]. Sources are piped into Resources as a flat list, then referenced into Projects and Areas. So Obsidian stays read-only; I don’t want the LLM editing notes I write by hand.

The wiki is scoped per project, not global. When you start something new, you reference that snapshot through the deep-research loop and scope it down to the project, running the loop or ingesting specific repos, articles, and notes via skills plugged into a harness.

Scoping is a scale choice as much as a safety one: plain files stay fast and inspectable at project size, but a whole-vault wiki of thousands of documents is exactly where a vector or graph database finally earns its place.

That’s the ~100-note wiki from the top, kept small by scoping to one project, yet reaching my entire 10,000+ note second brain through the loop.

The mental model is that the project is the work and the second brain is the research. A project is anything where you turn research into work: an article, a video, a set of slides (this talk was built this way), or a whole codebase. That’s the whole architecture, and the fastest way to see why it matters is to watch it run.

See It Run: Three Demos

Everything ships as one open-source Claude Code plugin you can clone and run on your own notes (https://github.com/iusztinpaul/ai-research-os-workshop), tweakable for any harness.

It’s four skills: /research (build or query the wiki), /research-distill (a per-piece research.md), /research-lint (health-checks), and /research-render (slides or a brief).

Demo 1 is research from a brain dump around “agentic harnesses”. You point the research skill at a file of notes plus a few must-include references. It scrapes those for seed context and picks a depth: fast (1 round of 3 queries), light (2 rounds, 3 then 2), or deep (3 rounds of 3). Out come the raw files + the LLM wiki:

Demo 2 is ingesting and comparing GitHub repos. You give it 3 harness repos (OpenCode, Pi, Hermes), no deep research, scoped to architecture, sub-agents, memory, and the permission flow. It clones each, writes per-repo notes, then builds cross-repo comparisons.

Demo 3 is ad-hoc links on GraphRAG, then questions to create notes. You hand it 3 arbitrary links to build a wiki, then ask how agentic GraphRAG differs from a plain knowledge graph. It answers off the wiki and updates it in place.

It was super hard to show the full demos here. You can find the full examples within the Examples section of the repo or watch me go over them in Obsidian and Zed in the full video.

Where This Is Going

Some connectors are missing on purpose (Google Drive, Notion, Slack) — the point is for you to add what you need. Louis-François added YouTube transcript ingestion in seconds with one prompt to Codex.

Two weaknesses stand out. The byproducts are often poorly written, fixable with the article writing profiles I already use. And as data grows, the LLM makes subtle errors: concept confusion (comparing Claude Code and OpenCode, it merged how each handles tool calls into one explanation) and superficiality (the Claude Code queuing system took many follow-ups to get right).

The fix is constant linting, which is what /research-lint does: it sweeps the wiki for orphan sources, broken links, stale claims, and contradictions.

What’s Next

Everything you saw here is open source. Clone the repo (or install it as a Claude Code plugin) and run it on your own notes today: https://github.com/iusztinpaul/ai-research-os-workshop

If you’d rather see the whole system in action first, watch the full talk and I gave at the AI Engineer World’s Fair:

And if you want to go beyond a workshop repo, such as building a deep-research-plus-writing multi-agent system from scratch and shipping it to production with AI evals and an observability layer on top, that’s exactly what we teach in our Agent Engineering: Building Multi-Agent Systems course.

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

Explore Next

1. Karpathy, A. (n.d.). LLM Wiki: A pattern for building personal knowledge bases using LLMs. GitHub. https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f

1. Google Cloud. (n.d.). How the Open Knowledge Format can improve data sharing. Google Cloud Blog. https://cloud.google.com/blog/products/data-analytics/how-the-open-knowledge-format-can-improve-data-sharing

2. Forte, T. (n.d.). The PARA Method: The Simple System for Organizing Your Digital Life. Forte Labs. https://fortelabs.com/blog/para/

Images

If not otherwise stated, all images are created by the author.

Models are commoditizing fast. Harnesses already have. What you care about is your research, your notes, your conversations, your tasks, your preferences, and your domain knowledge.

“Free” open-source harnesses don’t make you free. What you want to own is the context layer. Every agent is the same model + runtime + harness underneath, rebuildable on an open stack [1].

I hit this myself. The deeper I built into a single harness, the clearer it got how much I’d lose the day I had to leave. The loss comes in 3 forms.

Failure 1 — You start from scratch. You’ve run Claude Code for 6 months. You switch to an open model and every past conversation, every preference the agent learned about you, is gone. You begin again from zero.

Failure 2 — Your skills are hostage. The harder lock-in is your business logic. If your skills and workflows are coupled to one harness’s features and keywords, switching doesn’t just lose chat history — your custom logic breaks or quietly performs worse on the next tool.

Failure 3 — You’re billed at their mercy. The plan you depend on can be pulled, suddenly gated behind expensive pay-as-you-go, restricted from the latest models (e.g., the Fable story), or repriced from 200 to 1,000 overnight. With high switching friction, you can’t leave. You never know how the AI world can change. But for sure it won’t stay as is now.

So what do you do?

Keep your context layer detached from the harness. Any harness, any model (open or closed), plugs in and within ~5 minutes knows who you are, what you’re working on, what matters to you and how you like things to get done.

Everything that follows is based on the two projects I am working on: Scrabble, my own context layer anchored into my Obsidian notes, Readwise library, Notion, and Google ecosystem; and Tree, the project I am working on with for our next book, where we teach how to build a personal AI assistant from scratch.

The Architecture of Your Context Layer

A context layer is made out of 3 core components. First, a unified memory that holds everything you know. Second, a serving layer (MCP server or skills) that is the interface to that memory and the business logic that defines how the memory is used. The harness sits on top and is deliberately disposable.

The unified memory fuses what used to be separate systems: a filesystem, keyword search (BM25), semantic vector search, and a knowledge graph of typed POLE+O nouns (Person, Organization, Location, Event, Object), all in one place. The goal is a memory that belongs to you.

What I always preach is to try your best to build your unified memory on top of a single database that supports text, semantic, and graph search (such as MongoDB). Start simple and add complexity only when your use case demands it.

The serving layer has 2 angles. The first is an MCP server, surfacing Tools, Resources, Prompts, Skills, and even MCP Apps (e.g. a graph visualizer) [2]. It wraps the business logic for how memory is queried and updated. That’s what makes it portable.

The second form skips the server entirely, which is based on skills shipped straight on the filesystem (an AGENTS.md plus a folder of skills). Skills are an interface too. They’re leaner, with nothing to run, but more tied to a given harness’s conventions. It’s how Scrabble’s wiki layer gets consumed.

In practice the strongest setups do both: a portable MCP server for memory access, skills layered on top for higher-level workflows [2]. To avoid fragmenting your business logic, host the skills directly on your MCP servers, coupling them with the rest of the server’s business logic.

In Tree, where I am building a unified memory powered by knowledge graphs, everything is discovered as ordinary MCP tools. That is exactly why “swap the harness, keep the memory” is a one-line config change, not a migration.

Here is what happens when you interact with the context layer:

You enter a prompt into the harness, which becomes an MCP tool call to the server. The server queries the unified memory, and results flow back into the agent’s context. Nothing crazy so far.

The interesting part happens when you start adding stuff to your memory. It’s where continual learning happens.

Building a Unified Memory for Continual Learning

At a high level, the pipeline is simple. Ingest data into the unified memory → extract entities and relationships into a graph → index for hybrid and graph retrieval → expose everything through an MCP server → connect it to a harness like Claude Code or OpenCode. Building the knowledge graph was the easy part. Designing how the agent interacts with it was the hard part.

Don’t expose raw database operations. Agents struggle with them. Instead, design the server around how agents actually search and write, giving them high-leverage primitives they can compose like LEGOs inside skills. In Tree that’s 6 tools: 3 to search, 3 to write.

There are 3 search tools. nl_query_memory is the default — an LLM maps a natural-language (NL) question into a MongoDB hybrid-search + graph query. query_memory is the deterministic fallback for structured filters when NL fails. deep_search_memory handles large result sets (50+ docs) via progressive disclosure: it writes intermediate results to a YAML index creating a localized LLM wiki on the fly.

The distinction between nl_query_memory / query_memory and deep_search_memory is important because when retrieving chunks from the unified memory, you need to further compress them before adding them into the context window to avoid exploding your costs. But that means you lose a lot of information. Thus, creating a wiki on the fly trades latency for performance and lower costs.

There are also 3 write tools: ingest_url, ingest_file, and ingest_conversation.

The last is auto-triggered by a hook that ingests incrementally every ~10 conversation turns and finalizes when the conversation ends. That hook is what makes the learning loop continual: the system writes back what it just learned without you asking, so it gets smarter the more you use it.

And “smarter” is concrete. Across tens of thousands of notes, it finds links between things you’d long forgotten you had, connections you’d never have made by hand. It’s how you finally harvest your own past work and your graveyard of bookmarks.

Now, let’s see how to actually build the unified memory layer.

Build It Three Ways

You’re implementing 3 pieces: the unified memory, your custom business logic, and the serving layer. How much you build yourself is also split into 3 levels of effort, which translates to a build vs. buy discussion.

Level 1: Build all three from scratch on a database that supports everything at once (such as MongoDB), implementing the memory and the MCP serving layer yourself. Maximum control, maximum effort.

Level 2: Build only the business logic and serving layer, on top of an SDK that already implements the memory layer: Graphiti, a bi-temporal entity/fact/episode graph; Neo4j-Labs’ agent-memory, a graph-native POLE+O store; or mem0, a vector-first layer that stores facts as rows.

Level 3: Off-the-shelf memory engines you mostly just run: cognee, an Extract-Cognify-Load pipeline of typed nodes that are graph + vector at once; or managed services like Zep (temporal context graphs on the open-source Graphiti engine) and HydraDB (a graph DB on tiered object storage, pitched as “own your memory, no data leaving your stack”). Lowest effort, least control over the data model, and a vendor dependency. Cognee can import and export memory across Mem0, Zep, and Graphiti if you need to switch.

Now zoom into Level 1, the from-scratch path. Building the memory yourself splits into 2 versions. Version 1 (more complex), such as Tree, the agent memory I am building, which is a GraphRAG system whose retrieval fuses vector and keyword search with knowledge-graph traversal. It contains a POLE+O ontology, the index (graph + vector + text), the data and memory pipelines that ingest into it, and the query algorithms that read it.

POLE+O is a popular data model for building ontologies, borrowed from law-enforcement and intelligence analysis. Its beauty is that it’s perfectly balanced. It’s not too shallow, nor too deep. Perfect for an LLM to extract KG triplets from your data. Here it is as a thin StrEnum of the 5 node types:

In case you want more granularity over your POLE+O model, you can introduce subclasses, such as:

• Person: individual, alias, persona

• Organization: company, nonprofit, government, ...

• Location: address, city, region, country, ...

• Event: meeting, transaction, communication, ...

• Object: device, software, document, task, topic, project

To learn more, I have a full article on agentic GraphRAG and another one on designing an ontology for your context layer.

Version 2, the lighter one, is based on LLM wikis, such as Scrabble, my current agent memory. Markdown files with YAML frontmatter and cross-references, sitting directly on top of your existing infrastructure (Obsidian, Notion, Google Drive) — exactly how Scrabble runs over my own second brain. Google recently formalized this pattern as the Open Knowledge Format (OKF), a vendor-neutral directory of markdown + YAML whose only required field is type [3]. The filesystem is the state; no database required.

Which do you pick? It depends on how much control over the data model you want and your technical depth. Personally, I use both: Scrabble for everyday research across my second brain, and I plan to use Tree when I need precision at scale. Such as mining high-signal knowledge from 10,000+ documents.

Still, when it comes to implementing a unified memory layer on top of a knowledge graph, I keep getting asked one question: is a single database (MongoDB) enough? Why not throw in a specialized graph database like Neo4j or a vector database like Pinecone?

Is MongoDB Enough?

The simplest system that works wins. Instead of 3 databases (a SQL/NoSQL store, a vector DB, and a graph DB), you want one store that does all three. The payoff is concrete: far less operational overhead (one production DB, not three), less deep expertise required (being a power user of one engine beats being mediocre at three), and the ability to join documents, vectors, and graph in a single query. That last point matters most for memory: a single-store join is faster, cheaper, and lower-latency.

MongoDB is a good example, because it’s schema-less, supports all indexing operations, and has a huge ecosystem around it, with both an open-source version and their self-managed Atlas version. Another option I’ve tried that works well, but with more developer friction, is Postgres.

Also, as the cherry on top, you get lineage for free. Which is essential for adding references, understanding where each node came from, or simply preparing for an audit. You keep references on each knowledge-graph node instead of copying all data onto it: the document(s) a node was extracted from, the user who owns it, other metadata. That gives you easy lineage and versioning, and lets your memory connect to the rest of your system with no cross-database sync tax.

Now what about scale? For personal assistants, where you have only 100-10,000 documents, a single MongoDB cluster is more than enough. But even for medium to big enterprises, you can easily scale to millions of documents by adding more clusters (aka horizontal scaling).

After a conversation with a principal MongoDB engineer I understood that the real bottleneck is RAM. RAM is the most expensive component from your database cluster and you want to keep it as low as possible.

So index only what you query. Your memory holds 2 snapshots of the same knowledge: an append-only log (the immutable record of everything ingested) and a materialized view (the queryable graph rebuilt from that log). Vector indexes are inverted indexes ≥ the data they cover, so indexing both blows ~10 GB of data up toward ~40 GB of RAM.

Keep the log on disk (no vector index) and index only the materialized view, and it collapses back toward ~10 GB with the same data, at a quarter of the RAM.

When does it make sense to use a specialized graph DB like Neo4j? The reality is that for most use cases, you will do only 2-3 hops traversals. Such as getting all the preferences of a user, and the documents/conversations they were extracted from. In these cases, a single database that does it all is amazing. But there are cases when you should reach for a specialized graph DB like Neo4j. For example, when you need to do 3+ hop traversals, your whole business logic relies on graphs or simply as an internal visualization tool. You could sync your MongoDB production database to Neo4j, just for exploration reasons, as their Cypher engine + visualization ecosystem is stronger.

Using Your Context Layer With Any Agent

The wiki version is the simplest to switch between agents. If your context layer is shipped as an LLM wiki, switching harnesses is just handing the new harness a path. For this article, I literally pointed the harness at the research wiki folder from my Second Brain. Without any fancy skills or plugins in place.

Or a step further, is to point it at my entire second brain — that’s Scrabble — whose AGENTS.md explains how to navigate it and which CLI tools and skills to use. It works out of the box because it’s just files.

The MCP version is a bit more work, but not much. Serving memory over an MCP server means re-pointing one config entry from one harness to the next — switch from Claude Code to Codex by configuring the path to your MCP server, and your whole memory moves with almost zero friction, regardless of harness or model.

Each MCP server ships its own tools (read tools and write tools), and the tool descriptions are the contract the agent reads. It gets especially interesting on the write side, where the tools decide when to persist (e.g. parsing a whole conversation into memory when it ends).

Skills are where the leverage compounds. On top of this memory, you can write deep-research, writing, or coding-agent skills that remember your preferences. A few of mine, all part of Scrabble:

• /research, which runs deep research on a topic and produces a localized wiki (I have an open-source version in my latest ai-research-os-workshop talk made for the AI Engineer World’s Fair SF - full video);

• /article-guideline, which expands a brain dump into an article plan anchored in a wiki;

• /squid:plan, part of my Squid software factory, which plans a feature from my spec, codebase, and memory — injecting my personal takes on how I write software.

What’s Next

Agent memory is an unsolved, fast-moving topic. These designs work, but they’re far from perfect, and your data is still segregated across Obsidian, Notion, local files, and messages in ways that are hard to unify. In practice, I lean on the lighter wiki-based approach. One per project does the job. I reserve Tree’s heavy knowledge-graph memory for the few high-precision cases that earn its cost.

But here is what I’m wondering:

Your data can move between harnesses, but can your skills? I have this issue with Claude Code’s workflows and agents embedded into my skills.

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

Special thanks to MongoDB for sponsoring this article and keeping it free!

Explore Next

1. LangChain. (n.d.). Open Models, Open Runtime, Open Harness — Building Your Own AI Agent With LangChain and Nvidia. YouTube. https://youtube.com/watch?v=BEYEWw1Mkmw

2. Soria Parra, D. (n.d.). The Future of MCP. YouTube. https://www.youtube.com/watch?v=v3Fr2JR47KA

3. McVeety, S., & Hormati, A. (n.d.). Introducing the Open Knowledge Format. Google Cloud. https://cloud.google.com/blog/products/data-analytics/how-the-open-knowledge-format-can-improve-data-sharing

4. getzep. (n.d.). Graphiti. GitHub. https://github.com/getzep/graphiti

5. Neo4j Labs. (n.d.). agent-memory. GitHub. https://github.com/neo4j-labs/agent-memory

6. mem0ai. (n.d.). mem0. GitHub. https://github.com/mem0ai/mem0

7. topoteretes. (n.d.). cognee. GitHub. https://github.com/topoteretes/cognee

8. getzep. (n.d.). Zep. https://www.getzep.com/

9. HydraDB. (n.d.). HydraDB. https://hydradb.com/

10. Iusztin, P. (n.d.). ai-research-os-workshop. GitHub. https://github.com/iusztinpaul/ai-research-os-workshop

11. Iusztin, P. (n.d.). squid. GitHub. https://github.com/iusztinpaul/squid

Images

If not otherwise stated, all images are created by the author.

The scariest AI failures are the silent ones.

You ship a change to your agent on a branch — a new feature, a prompt fix, a quick refactor. No errors. No complaints. Everything looks fine. But does it still work, or did you quietly break something that worked yesterday?

As Alejandro Aboy puts it: “the fact that they’re not complaining doesn’t mean there’s no issue going on.” A quiet user is not a happy user. Usually, it’s the opposite.

The more Alejandro and I talked about AI evals and EDD, the more his struggles sounded like mine. A story from builders to builder.

1. You can break what already worked. Change a prompt, refactor a tool, and an old feature quietly regresses. Alejandro lived it: cleaning noisy instructions out of his agent’s system prompt made it start fabricating IDs it used to get right. You only catch that by running the same tests before and after the change, and comparing.

2. The feature is brand new, so you have nothing to test it on. No dataset, no historical traces, no ground truth. Yet you still need to know whether it works, and how well. So how do you generate realistic test data fast, then feed it to evaluators that turn it into hard performance numbers?

This is the case study that gives you a plan of attack for both: Evaluation-Driven Development (EDD)

How to prove a new feature works, and didn’t regress, before you merge. It comes from a recent conversation with , a senior data and AI engineer at Workpath who owns the entire data stack, built the Workpath AI Companion, and writes The Pipe and The Line Substack.

And we won’t keep it abstract. Every example comes from one real product: Workpath, a strategy-execution SaaS that keeps large companies’ OKRs and initiatives aligned. (OKRs — Objectives and Key Results — are the goal-setting framework teams use to name what they want to achieve and the measurable results that prove they’re getting there.) Its AI-native feature is the Workpath AI Companion: an agent that scans a company’s strategy and OKR data end-to-end to keep enterprise teams aligned. It’s the exact system Alejandro runs EDD on every day.

So when we say developing a new feature, picture shipping a change to that Companion and proving, before you merge, that it works and didn’t regress.

The Develop-a-Feature Workflow

Imagine. You start working on a new feature, branch, and develop the change. But before you merge, you have to answer 2 questions:

1. What’s the performance of my new feature?

2. Did my change introduce any regressions into the existing codebase?

Only when both look good you accept the pull request. EDD helps you answer those 2 questions.

Every feature is hypothesis-first. As Alejandro frames it, “I have a hypothesis... and everything should lie around that.” Every change starts as a stated hypothesis on a branch.

Based on that hypothesis, EDD runs a simulation and scores the results to answer the 2 questions.

Every feature ends in a PR, backed by an experiment, with clear traces and metrics. Framing the eval results as an experiment allows you to compare current results to previous ones, detecting regressions or tracking improvements.

This is how you can compare two experiments in Opik: