Every AI application that wraps an agent is a harness!

In LangChain’s Terminal-Bench experiment, changing only the harness (with the same model) moved a coding agent from ~30th place into the top 5: the harness, not the model, is what makes a coding agent good.

In the open-source course Building a Coding Agent From Scratch, you’ll build that harness from scratch in Python: Decode, a complete coding agent that grows lesson by lesson from a bare agent loop into a swarm of remote agents running in parallel in the cloud.

Why? You’ll be able to engineer custom harnesses for your own AI products (the skill behind that leaderboard jump), and you’ll understand what Claude Code and Codex actually do under the hood, turning you into a power user.

Lessons:

1. Building a Coding Agent From Scratch

2. The Bare-Bones Coding Agent Loop

3. From a Raw Shell to a Sandboxed Coding Agent

4. Context Engineering for Coding Agents ← you are here

5. Agents Catalog, Subagents & Parallel Fan-out ← Available next week

6. Remote Headless Mode & Durability

7. AI Evals Foundations: Benchmarks, Regression and Online

8. AI Evals on Steroids via Replays

Full open-source course

Lesson 4: Context Engineering for Coding Agents

I got hooked on running Claude Code agents 24/7. Getting things done while answering emails, cooking, or watching a movie. Until my subscription maxed out mid-turn with the agent halfway through a feature. Sounds familiar?

Complaining to your employer, buying a bigger subscription, changing the model, or switching harnesses only treats the symptom. Doesn’t solve the root cause. You spend money while the window stays noisy, degrading output no matter whose model is behind it.

The actual solution is to better understand harnesses and the context engineering behind them. Improve planning, develop stronger skills and memory, and know when to drop your context.

So far in the course, we have focused on harness engineering and building a sandboxed agent loop. Now it’s finally time for some context engineering for coding agents: memory, skills, LSP servers and compaction.

The whole problem resolves to what to put into context, what not to put, how to trim it down, plus creating as many feedback loops as possible for the agent, as Anthropic frames it as finding the smallest possible set of high-signal tokens.

You will walk away understanding and building from scratch:

• What the agent should carry between sessions.

• How skills load nothing until needed.

• The cheapest feedback loop in the system.

• How the window gets trimmed before it rots.

The context lifecycle of a session

To see how the 4 components cooperate, look at the demo-5-sandbox-feature-pr skill from the course repo (.decode/skills/demo-5-sandbox-feature-pr), where we encoded a demo where Decode writes a new feature into Decode by spinning up a background sandboxed session, where the host agent acts as the orchestrator and the sandboxed one as the feature executor. The final artifact from the demo will be a PR containing the new feature.

Go to repository, run decode, type /demo-5-sandbox-feature-pr, and enter. Let Decode do the rest of the work.

Here is a snapshot of the skill, where Decode spawns another Decode subagent:

...

## 1. Launch decode against a sandboxed clone of the course repo

Launch the local Docker run (Docker must be running):

"""
SANDBOX_MODE=docker decode --repo git@github.com:decodingai-magazine/building-a-coding-agent-from-scratch-course.git
"""

...

At session start, the system prompt is assembled from four parts: the base prompt, the active agent’s prompt, memory files (AGENTS.md + .decode/MEMORY.md), and the skills catalog (one line per skill). Pydantic AI adds each tool’s schema, paying off Lesson 2’s warning that every tool costs tokens. This assembled prompt enters context, shaping the probabilities of the model’s next steps.

The run fills the window. Invoking the skill loads its SKILL.md body (tier 2 of progressive disclosure). Plan mode dumps repo files as read outputs. During the build loop, each edit lands, the Diagnostics Enricher appends type errors for free, and bash runs tests until green output signals completion.

Now, from the context window point of view, what happens within the harness?

As in the image below, as soon as we open the agent, the context window is filled with its system prompt, tool and skill descriptions and memory files. After invoking the /demo-5-sandbox-feature-pr skill, it gets filled with tool inputs and outputs, plus the SKILL.md file containing the specific instructions.

Next, through progressive disclosure, the agent begins reading the relevant files and scripts associated with the skills. Ultimately, it starts writing new Python files or editing existing ones that are statically checked for syntax issues via our ty Language Server Protocol (LSP) server.

On Modal’s self-hosted Qwen3.6-35B, the window is 262,144 tokens. Usually, microcompaction fires at 60%, while full compaction is at 80%. After compaction, usage drops back to roughly 5–10%, so the session continues instead of crashing.

In the video below, you can see part of the 219 spans trace in Opik, clearly monitoring the LLM tool calls, token counts, and latency of the harness and LLM calls:

Every component in this lifecycle has specific mechanics: memory, skills, the LSP server, and compaction. Let’s explore each.

Memory: Stop repeating your instructions

In my early agent runs, the agent kept writing naive datetime objects instead of timezone-aware ones and added type hints inconsistently, forcing me to retype the same corrections session after session. The fix was to write the preference down once into the AGENTS.md, where the agent reads it every turn.

AGENTS.md injects project context: business logic, why components exist, the tech stack, and the processes around it (docs, deploy, review, testing). The code is the source of truth, so avoid duplicating it. Add metadata and references the agent can discover without heavy reasoning.

Keep it under 300 lines, with a guardrail of around 600 lines. Write each line in response to an observed mistake so the agent avoids repeating it, following Mitchell Hashimoto’s rule.

Decode recursively looks within all the project directories for AGENTS.md files (root-most first, so the nearest file takes precedence), appends .decode/MEMORY.md last, stamps each with a # From <path> provenance header, and dumps the result into the system prompt.

If AGENTS.md is what you manually define, .decode/MEMORY.md is what the agent automatically extracts from your conversations, replicating Claude Code’s auto-memory

At the end of each session — on quit and on /clear — one cheap LLM call summarizes the session into a single plain sentence, appended as a dated bullet (- 2026-06-26: …), as an append-only log. As this can grow big fast, the file has a hard cap of 200 lines or 25,000 bytes, dropping the oldest lines first.

From src/decode/memory/extract.py:

async def extract_on_exit(messages: list[ModelMessage], cwd: Path) -> None:
    summary = await summarize_session(messages, model_or_settings=settings)
    append_session_summary(cwd, summary, now=_utc_now())

    if settings.memory_compression_enabled:
        await compress_memory_file(cwd, model_or_settings=settings)

summarize_session is the single LLM call that distills the conversation into one sentence (or None if it’s not worth saving). append_session_summary writes that sentence into .decode/MEMORY.md and enforces the hard cap.

compress_memory_file runs Memory Compression to rewrite the file in place, merging duplicate or superseded notes while preserving dated bullets.

Skills: Never load what you can reference

My Python testing and PR conventions used to live directly in my memory file, bloating the context of every session. Moving them into skills left behind just a few lines, as references, that specify when to access each one.

Skills prevent context rot from 2 directions. On the tools side, upfront schemas burn budget: Mario Zechner measured that popular MCP servers consume 7–9% of the context window before any work begins. On the memory side, stuffing review guides and workflow templates into AGENTS.md pollutes every turn. Skills solve both: each phase-specific behavior lives in its own skill, referenced from a one-line catalog entry, and loads only when that workflow phase runs.

Skills follow the Agent Skills standard. In Decode, skills live in .decode/skills/, or you can open the TUI, type /, and pick one. A separate public registry lives at skills.sh (npx skills install <skill>):

my-skill/
├── SKILL.md          # Required: metadata + instructions
├── scripts/          # Optional: executable code
├── references/       # Optional: documentation
├── assets/           # Optional: templates, resources
└── ...               # Any additional files or directories

Progressive disclosure operates across 3 tiers. In tier 1, only the skills catalog stays in context: one name + description line per skill. As your library grows, an optional guard can cap the catalog at ~1% of the context window.

A common strategy Decode doesn’t have yet is user-invocable-only skills, where you flag a skill as callable only. This means you can remove its name and description from the catalog, leaving it with zero context until explicitly invoked by the user.

In tier 2, invoking a skill via the skill dispatcher tool or /<skill-name> loads only its SKILL.md body and any other files packed within the skill.

From src/decode/tools/skills.py:

async def skill(ctx: RunContext[AgentDeps], name: str) -> str:
    home = ctx.deps.harness_home or ctx.deps.cwd
    catalog = load_skills(home)
    found = catalog.get(name)

    return format_skill_payload(found, cwd=home)

In tier 3, because within format_skill_payload we properly format and expose all the available resources (files, scripts, docs, assets) from a skill to the agent, if it considers them necessary, it will load them via its read tool or execute the containing scripts via the bash tool.

This is the core idea of progressive disclosure. It’s mostly just exposing a manifest of bundled files with exact cwd-relative paths, making it super clear to the agent how to access them. The key here is that the LLM is properly trained for tool calling to make the right decisions about whether to call read or bash tools.

From src/decode/skills/payload.py:

def format_skill_payload(skill: SkillDef, *, cwd: Path) -> str:
    if skill.resource_dir is None:
        return f"{skill.body}\n\n{_OUTPUTS_TRAILER}"
    rel_dir = os.path.relpath(skill.resource_dir, cwd)
    files = _bundled_files(skill.resource_dir)
    
    listing = "\n".join(f"- {rel_dir}/{name}" for name in files)
    trailer = (
        f"Bundled files for this skill (all under `{rel_dir}/` — use these EXACT paths):\n"
        f"{listing}\n"
        "Read them with the `read` tool; run `scripts/` files with `bash`."
    )

    return f"{skill.body}\n\n{trailer}\n\n{_OUTPUTS_TRAILER}"

Now let’s look at the cheapest feedback loop in the whole coding agent.

The LSP server: Replace guessing with precision

In my multi-agent setup, each turn between the engineer and tester agents re-ran the linter, type checker, formatter, and test suite. My Prefect orchestrator integration tests took 15 minutes on their own, so I split them to speed up the feedback loop.

The takeaway was clear: the most important part of your agentic flow is to always give as many feedback loops as possible. The LSP server is the fastest way to feed in code-related signal.

An LSP server is one of the most underrated components, particularly for coding harnesses. It maintains a live index of symbols across your codebase: variables, functions, classes, definitions, references, and type errors. Your IDE already runs one per language. You need one for each programming language. Decode uses ty by Astral, an extremely fast type checker and language server written in Rust, to support Python. Made by the same guys behind uv and ruff.

The server feeds the agent signal through 2 channels.

Channel 1 is the lsp tool, which handles active queries with 4 ops: definition, references, hover, and diagnostics. One call returns a precise file:line:column answer instead of 3 speculative file reads. It’s read-only, so the permission gate auto-allows it across all modes.

From src/decode/tools/lsp.py:

async def lsp(
    ctx: RunContext[AgentDeps],
    op: str,
    path: str,
    line: int | None = None,
    column: int | None = None,
) -> str:
    if op == "definition":
        return await _run_definition(ctx, path, line, column)
    if op == "references":
        return await _run_references(ctx, path, line, column)
    if op == "hover":
        return await _run_hover(ctx, path, line, column)
        
    return await _run_diagnostics(ctx, path)

async def _run_definition(ctx: RunContext[AgentDeps], path: str, line: int, column: int) -> str:
    result = await lsp_service.definition(ctx.deps.cwd, path, line, column)

    return _format_location(result)

The server runs as a background process at each project root, communicating via JSON-RPC (a plain request/response protocol over standard input/output). Decode’s LspClient initializes the session, negotiates capabilities, and sends requests:

From src/decode/services/lsp/service.py:

from decode.services.lsp.client import LspClient


process = await asyncio.create_subprocess_exec(
    "ty", "server",
    stdin=PIPE, stdout=PIPE, cwd=root,
)
client = LspClient(process, root)
await client.initialize()

In the demo-5 session, the agent resolves the entry point via lsp("definition", "src/decode/cli.py", <line>, <column>) over JSON-RPC textDocument/definition. On the next turn, the model targets its edit tool call at that exact location, rather than blindly exploring the codebase first.

Channel 2 is the Diagnostics Enricher, which runs passively on every successful .py write or edit to append an errors-only block to the tool result. It displays at most 10 errors, and stays silent on clean files or when the server is unavailable, matching the pattern in OpenCode.

_enrich wraps the return value of file modification tools without requiring extra turns or tools.

From src/decode/tools/files.py:

def _enrich(base: str, cwd: Path, path: str) -> str:
    summary = _format_lsp_errors(lsp_service.diagnostics_on_edit(cwd, path))
    if summary is None:
        return base
    return f"{base}\n\n{summary}"

def _format_lsp_errors(diagnostics: list[Diagnostic] | None) -> str | None:
    ...
    errors = [d for d in diagnostics if d.severity == _LSP_ERROR_SEVERITY]
    shown = errors[:_LSP_DIAGNOSTICS_LIMIT]
    lines = [f"LSP diagnostics ({settings.lsp_server_command}) — fix these:"]
    ...
    return "\n".join(lines)

In the demo-5 session, when the agent updates src/decode/cli.py with an unimported reference, the file writes successfully, but the enricher appends LSP diagnostics (ty) — fix these: ... with the error details. The model sees this feedback immediately and fixes the import on the next edit before running any tests.

In the video below, you can clearly see in Opik how the LLM outputs LSP tool calls and how they are executed in the harness:

Memory, skills, and the LSP all shape what enters the window. Now let’s see how we can trim it down.

Compaction: Delete before the window rots

Back in Nov 2025, when building the writing agent for my agent engineering course, requests started degrading around 180,000 input tokens on Gemini Pro, taking>3 minutes per request or directly returning timeout errors and disconnections. Considering that on paper Gemini handles up to 1M input tokens.

A full window degrades model performance and reliability long before hitting the hard token ceiling, following the degradation curves documented by Anthropic. That’s why you need compaction to continually reduce your context window while minimizing context loss.

Compaction handles this in 3 ways. The simplest is /clear, which wipes the entire window after running the on-exit memory write-back so key learnings persist in .decode/MEMORY.md.

The second option is full compaction, triggered automatically at 80% capacity or manually via /compact. An LLM summarizes the conversations into a six-part template (goal, constraints & preferences, progress, key decisions, next steps, critical context), and older messages are dropped. The window becomes [system prompt] + [summary] + [recent tail], where the tail retains ≈20,000 tokens of recent messages snapped cleanly to a Compaction Boundary so tool calls remain paired with their results, matching Pi’s implementation.

Both tiers evaluate should_compact using the provider’s reported token window against reserve thresholds (80% full compaction ->20% empty, 60% microcompaction -> 40% empty):

From src/decode/context/compaction.py:

def should_compact(usage: RunUsage, *, window: int, reserve: float, enabled: bool) -> bool:
    if not enabled:
        return False
    if usage.input_tokens <= 0:
        return False
    return usage.input_tokens >= reserve_threshold(window, reserve)

Everything happens within the compact() method from AgentTurnHandler. split_tail walks backward through message history estimating token counts to locate the Compaction Boundary. summarize_for_compaction generates the summary, build_summary_message wraps it as a synthetic user message that is itself part of the history the next compaction summarizes (so successive compactions merge for free), and the handler sends [summary_message, *tail] on the next loop iteration. The harness owns the list it feeds the model, so replacing the list IS the compaction:

From src/decode/agent/loop.py:

class AgentTurnHandler:
    ...

    async def compact(self) -> CompactOutcome:
        split = split_tail(
            self.message_history, keep_recent_tokens=settings.compaction_keep_recent_tokens
        )
        skeleton = await summarize_for_compaction(
            self.message_history, model=self._compaction_model
        )

        before_tokens = self._last_input_tokens
        summary_message = build_summary_message(skeleton)
        tail = self.message_history[split:]

        self.message_history = [summary_message, *tail]

        return CompactOutcome.COMPACTED

The 3rd and last option is microcompaction, which runs without LLM calls at 60% capacity. It replaces tool outputs outside the recent tail with a placeholder string by inspecting each ToolReturnPart (the message part that holds a tool’s output). Because tool outputs are consumed each turn, conclusions live on in subsequent messages. Nothing is lost because the tool input remains available in the messages, allowing the agent to rerun the tool if necessary. That is why Anthropic calls tool-result clearing the safest, lightest touch of compaction.

After every completed turn, the handler checks the same number against both thresholds: full compaction first, then microcompact, replacing message_history through the same reassignment.

From src/decode/context/compaction.py:

_MICRO_PLACEHOLDER = "[tool output elided by microcompaction]"

def microcompact(
    messages: list[ModelMessage],
    *,
    keep_recent_tokens: int,
    placeholder: str = _MICRO_PLACEHOLDER,
) -> list[ModelMessage]:
    boundary = split_tail(messages, keep_recent_tokens=keep_recent_tokens)

    new_messages: list[ModelMessage] = []
    for index, message in enumerate(messages):
        if index >= boundary or not isinstance(message, ModelRequest):
            new_messages.append(message)
            continue

        new_parts = list(message.parts)
        changed = False
        for position, part in enumerate(message.parts):
            if not isinstance(part, ToolReturnPart | RetryPromptPart):
                continue
            new_parts[position] = dataclasses.replace(part, content=placeholder)

        new_messages.append(dataclasses.replace(message, parts=new_parts))

    return new_messages

When saving the session to JSONL files, there is no compaction. The file is a simple snapshot of the entire message history. To save a compacted session, you either have to run it before exiting the session or resume, run it and then exit.

In the image below, you can see how the /compact command reduced the context window from ◑ 57% (~149539) to 8% of the context window.

Next steps

There are other components we haven’t touched on in this article, such as an MCP client or an auto-mode permission layer. Still, these are the 4 harness components that are omnipresent in every coding harness you will use. Maybe Pi, with its minimalist design, is the only exception.

Within the next lesson, we will add the final harness piece to the puzzle: creating an agent’s catalog used to fan out subagents whose work never pollutes your window.

Here is the course roadmap, lesson by lesson (see all in GitHub):

1. Building a Coding Agent From Scratch

2. The Bare-Bones Coding Agent Loop

3. From a Raw Shell to a Sandboxed Coding Agent

4. Context Engineering for Coding Agents ← you are here

5. Agents Catalog, Subagents & Parallel Fan-out ← Available next week

6. Remote Headless Mode & Durability

7. AI Evals Foundations: Benchmarks, Regression and Online

8. AI Evals on Steroids via Replays

But here is what I’m wondering:

When your coding agent’s window fills up mid-task, what do you actually do today: /clear and lose the thread, /compact and hope the summary gets the job done, or just keep going until it degrades?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

Special thanks to Modal, Opik (by Comet), and Kitaru (by ZenML) for sponsoring this open-source course and keeping it free!

Images & videos

If not otherwise stated, all images are created by the author.

In LangChain’s Terminal-Bench experiment, changing only the harness (with the same model) moved a coding agent from ~30th place into the top 5: the harness, not the model, is what makes a coding agent good.

In the open-source course Building a Coding Agent From Scratch, you’ll build that harness from scratch in Python: Decode, a complete coding agent that grows lesson by lesson from a bare agent loop into a swarm of remote agents running in parallel in the cloud.

Why? You’ll be able to engineer custom harnesses for your own AI products (the skill behind that leaderboard jump), and you’ll understand what Claude Code and Codex actually do under the hood, turning you into a power user.

Lessons:

1. Building a Coding Agent From Scratch

2. The Bare-Bones Coding Agent Loop

3. From a Raw Shell to a Sandboxed Coding Agent ← you are here

4. Context Engineering for Coding Agents ← Available next week

5. Agents Catalog, Subagents & Parallel Fan-out

6. Remote Headless Mode & Durability

7. AI Evals Foundations: Benchmarks, Regression and Online

8. AI Evals on Steroids via Replays

Full open-source course

Lesson 3: From Raw Shell to a Sandboxed Coding Agent.

Mid-session, Claude Code was running inside my Obsidian Second Brain when it fired off a cleanup command that deleted half my notes. If I hadn’t been backing them up with Obsidian Sync, two years of work would have been gone.

Even if you carefully isolate your agent, it can still reach the internet and go off the rails. In July 2026, OpenAI’s agents hacked Hugging Face. A week later, Anthropic disclosed that, after analyzing 141,006 eval runs from an isolated harness, their Claude models had gained unauthorized access to production infrastructure across 3 real organizations.

Still, for normal people who care about protecting their data, sandboxing is THE containment solution.

In Lesson 2, we saw how to implement an agent loop that controls your computer through 4 core tools: read, write, edit, and bash. In this article, we will learn how to isolate every computer-use tool from the rest of your system inside a sandbox.

We will hook it to two types of sandbox backends — local Docker and remote Modal — plus explore the other options and their trade-offs.

The cherry on top? GPU compute and scaling out of the box, plugged straight into your harness as a control center.

By the end, you will run:

SANDBOX_MODE=modal decode --repo https://github.com/<your-repo>.git "Swap from Gemini to Kimi K3."

Which will spin up a remote Modal sandbox hooked to Decode — the educational harness we are building throughout this course — set up the given repo, implement the requested feature, and end with a PR for review as the final artifact.

Try it yourself

The tools you love already use a sandbox

Locally, every bash command Claude Code or Codex CLI runs is wrapped in an OS-level jail — Seatbelt on macOS, bubblewrap on Linux (kernel features that block a process’s filesystem reach and syscalls).

In the cloud, Codex isolates every task in its own environment (aka sandbox) preloaded with your repo.

In that case, what’s a sandbox? It’s an execution boundary: the agent runs every tool that can alter the host inside a “jail”, so a wrong command runs inside a container, not your host.

Ok… That’s abstract. So how does a harness actually run its tools in this “jail”?

How do sandboxes actually work?

In Lesson 2, we learned that the core tools the agent uses to interact with your computer are read, write, edit, and bash. The rest (web_fetch, todo_write, ask_user, plan mode) never touch the filesystem — they are meta tools for fetching context and planning. That’s why, following Pi’s philosophy, they are optional.

Thus, our problem reduces to isolating the execution of the core tools from the rest of the harness. There are two main approaches.

In option 1, we run the whole harness in a Docker container or a remote Modal sandbox. It is straightforward and gives complete isolation, but it forces you to work in an environment different from your machine, with little flexibility to isolate specific tasks.

In option 2, we run the computer-use tools in a sandbox while the harness and the rest of the tools stay on the host. You keep your harness as your control center while executing tools inside the sandbox.

Option 1 is as simple as SSH-ing into a remote machine and running claude. Option 2 is where the real harness engineering happens.

The second dimension we have to think about is where the sandbox runs: locally in Docker or remotely on Modal.

Regardless of where the sandbox runs, every computer-use tool call gets wrapped in an inSandbox(command) call. In our educational coding harness, Decode, we defined a CommandExecutor interface with 2 implementations (LocalExecutor for the host, SandboxExecutor for a backend), powered by 2 sandbox backends: DockerBackend or ModalBackend.

From src/decode/sandbox/__init__.py:

def select_executor(mode: str) -> CommandExecutor:
    if mode == "docker":
        return SandboxExecutor(DockerBackend())
    if mode == "modal":
        return SandboxExecutor(ModalBackend())
    from decode.tools.exec import LocalExecutor

    return LocalExecutor()

The tool never knows where the command runs. The LLM emits the command, while the harness takes care of executing it in the selected environment. Once we build the right executor, we just call executor.run(command), completely abstracted away from the sandbox — which means we can extend it with sandboxes beyond Docker or Modal.

From src/decode/tools/bash.py:

async def bash(
    ctx: RunContext[AgentDeps],
    command: str,
    timeout: float | None = None,
) -> str:
    if needs_approval(ctx):
        raise ApprovalRequired  # Permission gate — before anything runs

    if not command.strip():
        raise ModelRetry("command is empty; provide a shell command to run.")
    timeout_s = _resolve_timeout(timeout)

    executor = await _get_executor() # sandbox (powered by Docker or Modal) or local
    result = await executor.run(command, cwd=ctx.deps.cwd, timeout_s=timeout_s)

    return _render(result, timeout_s=timeout_s)

We adopt a similar strategy for the read, write, and edit tools, plus optional ones such as glob and ls. That way the agent sees a single filesystem: when write creates a file, bash sees it immediately.

Now, let’s zoom in on how local sandboxes work via Docker.

Local sandboxes via Docker

Run “SANDBOX_MODE=docker decode --repo git@github.com:you/project.git" to start a new Decode session inside a Docker sandbox, which contains 5 main steps:

1. DockerBackend launches one long-lived keeper container running sleep infinity.

2. Injects all the environment variables from .env into the container.

3. Attaches a local volume at .decode/sandbox.

4. Prepares the Workspace by cloning --repo into .decode/sandbox at HEAD.

5. Installs the dependencies by running uv sync on the given --repo.

Within the DockerBackend class, plugged into SandboxExecutor, we have 2 functions to implement: create and exec. create mostly goes through the 5 steps outlined above. In exec, each bash call translates to a docker exec <command> against its associated container.

From src/decode/sandbox/docker_backend.py:

_WORKSPACE = "/workspace"  # container-side path of the Workspace

class DockerBackend:
    async def create(self, workspace: Path) -> None:
        # once, at session start
        # `workspace` is the host-side clone of your
        # --repo at .decode/sandbox

        args = ["run", "-d", "--rm", "-v", f"{workspace}:{_WORKSPACE}", "-w", _WORKSPACE]
        if sandbox_git_token():
            args += ["-e", GIT_TOKEN_ENV]
        args += ["ghcr.io/astral-sh/uv:python3.12-bookworm-slim", "sleep", "infinity"]
        proc = await asyncio.create_subprocess_exec(
            "docker", *args,
            stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE,
            env=_run_env(), 
        )
        stdout, _ = await proc.communicate()
        container_id = stdout.decode().strip()  # `docker run -d` prints the container id

    async def exec(self, *args: str, timeout_s: float) -> ExecResult:
        # for every bash tool call — a fresh exec
        proc = await asyncio.create_subprocess_exec(
            "docker", "exec", "-w", _WORKSPACE, container_id, *args,
            stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE,
            start_new_session=True,  # own process group → kill as a unit on timeout
        )
        ...  # gather stdout/stderr → ExecResult

Docker is easy to set up, works out of the box with container tooling, and the same containers can later be hosted remotely via Kubernetes or other orchestrators. As good as that sounds, it isn’t truly secure. Container processes are native processes on your kernel. As Abhishek Bhardwaj, on OpenAI’s RL and agent-infrastructure team, puts it, a container process can exploit that boundary and take the host — a kernel exploit is “a New York Times article waiting to happen”.

Docker sits on a spectrum:

1. fork/exec — straightforward to implement, no boundary. The command talks straight to your kernel.

2. Containers ← we are here. A namespace-and-cgroup boundary. Shared kernel. Another option is Podman, which doesn’t use a daemon, saving latency.

3. gVisor — a user-space “sentry” kernel answers the syscalls, turning a direct kernel exploit into a two-hop chain (sentry → host kernel). Costs little: near-container performance. It’s what Modal runs underneath its sandboxes. Not perfect. If the agent gets past the sentry, you’re back on the shared kernel.

4. microVMs — Firecracker or Cloud Hypervisor on KVM, the Linux kernel’s own hypervisor (Linux hosts only). The guest kernel runs in a separate CPU execution context from the host, so even if the agent fully compromises it, it can’t reach yours. Cheaper than it sounds: Arrakis boots one in under 7s, against ~40s for a traditional VM.

Seatbelt and bubblewrap sit on the same layer as containers. They are OS jails. They wrap one command instead of the whole machine, deriving the filesystem profile from the permission rules the agent already uses. No image, no daemon. Still sharing your host’s kernel. That’s what Claude Code and Codex CLI run locally. Cheaper than Docker, not stronger.

So which one? If you trust the code and just want your own files safe, go with containers (as we did in Decode). Otherwise, for full isolation, go with microVMs. Bhardwaj’s verdict, after building OpenAI’s sandbox cloud: “in the end, everyone always wants a VM… let me save you the story and two years of grief, just please use microVMs from the start”.

To run it on your own machine, follow the “Running the Code” setup steps in the course repo, then launch:

SANDBOX_MODE=docker decode --repo https://github.com/decodingai-magazine/building-a-coding-agent-from-scratch-course.git

This clones the repo into the isolated Workspace (/workspace ≡ host .decode/sandbox) and opens a new TUI session wired to the Docker container. To test it out, pick any feature, plan it, ask for a PR, and let Decode do the rest.

For a quick test, we prepared a demo wrapped as the /demo-5-sandbox-feature-pr skill. It uses Decode to spin up a new sandboxed Decode session and instructs it to implement a small feature from a pool of available ones (e.g. a decode --version CLI command), then open a PR with it. Creating feature PRs is essential when working in sandboxes, as you have no direct access to where the code actually runs.

The real win, though, is when the box isn’t on your machine at all.

Remote sandboxes via Modal

Modal‘s core primitive is the sandbox — an isolated, serverless runtime that starts in under half a second.

Run “SANDBOX_MODE=modal decode --repo git@github.com:you/project.git" and here is what happens through Modal’s 5-event lifecycle:

1. Created: ModalBackend requests a sandbox under decode-sandbox-<env>.

2. Scheduled: Modal finds capacity on its infra.

3. Started: the container is live and can execute commands (but the app is not ready yet).

4. Ready: the tar containing the app files is uploaded into /workspace (making the app env ready).

5. In use: bash and the other computer-use tools exec successfully against the remote.

Modal’s sandbox infra boots fast. Preparing the app dependencies is what takes a while.

That’s why, as you can see in the image below, we mount a volume that already contains the app dependencies, so they are ready when the container starts. On top of that, to avoid the cold start problem — you want the sandbox ready as soon as the harness starts — we prepare a sandbox pool modeled as a queue. We populate it with application-agnostic sandboxes and turn them into application-specific ones by attaching a volume. This combination of sandbox pool plus volume mounting gives us application-ready sandboxes on demand. More on this here.

Check this blog post if you want to learn more about how Modal dropped Kubernetes and built its sandbox architecture from scratch to start 1 million concurrent sandboxes in under a minute.

The ModalBackend class looks similar to the DockerBackend one. Inside create, we create the sandbox from a pre-built uv Docker image. We inject a git token to get access to our private repositories. Finally, we launch it with the sleep infinity command to keep it running — now on Modal’s infrastructure, which takes care of our security concerns.

Modal puts zero risk on your host. It’s remote, and it runs gVisor to intercept syscalls before they reach the kernel. Even when OpenAI’s agents hacked Hugging Face, as reported here, their code was running on one of Modal’s sandboxes — the agent exploited Hugging Face’s code running there, not the Modal infrastructure itself.

Within the exec method, we execute the command that comes from the LLM and return the output and exit code.

From src/decode/sandbox/modal_backend.py:

class ModalBackend:
    async def create(self, workspace: Path) -> None:
        app = await modal.App.lookup.aio(_app_name(), create_if_missing=True)
        image = modal.Image.from_registry(
            "ghcr.io/astral-sh/uv:python3.12-bookworm-slim"
        ).apt_install("git", "curl", "ca-certificates")
        secrets = []
        if token := sandbox_git_token():
            image = image.run_commands(GIT_CREDENTIAL_HELPER)
            secrets = [modal.Secret.from_dict({GIT_TOKEN_ENV: token})]
        sandbox = await modal.Sandbox.create.aio(
            "sleep", "infinity", app=app, image=image,

        ...  # tar-upload the Workspace into /workspace — the "Ready" step

    async def exec(self, *args: str, timeout_s: float) -> ExecResult:
        proc = await sandbox.exec.aio(*args, workdir=workdir, timeout=timeout, text=False)
        stdout, stderr = await asyncio.gather(proc.stdout.read.aio(), proc.stderr.read.aio())
        exit_code = await proc.wait.aio()

        ...  # → ExecResult

From Modal’s pricing page at the time of writing: a CPU sandbox at 2 cores + 4 GiB runs ≈ 0.38/hr, a B200 GPU bills ≈6.25/hr, and an H200 ≈ $4.54/hr. For pure agentic work, the CPU sandbox gets the job done, while the GPU ones let you run inference or fine-tuning jobs directly from your harness.

The default is using the CPU sandbox. To configure it with x4 H200 we would do:

sandbox = await modal.Sandbox.create.aio(
    ...,              # Same parameters as the CPU sandbox
    gpu="H200:4",     # 4× H200 attached to this sandbox
)

As discussed in Lesson 2 on pay-per-token vs. serverless, for ad-hoc data processing serverless can come out ~80-90% cheaper.

The trade-offs of a remote sandbox over a local one: network latency, sandbox management, and extra costs.

You can run the same test as for the Docker sandbox by following the “Running the Code” Modal extra setup steps and swapping SANDBOX_MODE=docker for modal:

SANDBOX_MODE=modal decode --repo https://github.com/decodingai-magazine/building-a-coding-agent-from-scratch-course.git

After running it, Modal’s dashboard shows how many sandboxes are live (5 in our case):

And if you open the sandbox app and click “Sandboxes”, you get the full list:

These sandboxes are created when you enter a new Decode session and automatically cleaned up when you exit via the /quit command.

The wanted side effects of remote sandboxes

Remote sandboxes give you 2 powerful side effects beyond safety.

The first is compute. Modal sandboxes accept a GPU spec like gpu="B200:8" at creation, letting you agentically fine-tune models or process large datasets (eg extracting knowledge graphs from 1000+ documents) with open-weight LLMs such as Qwen3.6, Gemma 4, K3, or GLM5.2.

The second is scale. An orchestrator agent running directly on the host (no sandbox) can hand work to background agents running on Modal remote sandboxes. Put them on a CPU if you just need a bunch of parallel agents hitting an LLM API to pull tickets from your Linear backlog, or on a GPU if you need to squeeze out more juice.

In this case study, Ramp, a fintech company, runs every agent session in its own Modal sandbox with a full dev environment inside. Modal’s own conclusion from the case study is that with cheap isolation, the bottleneck shifts from “can the agent write correct code” to “how many agents can you run in parallel”.

Next steps

Should you sandbox all the time? No. As a Claude Code power user, to keep it simple, I still run directly on my machine in folders versioned by git or Obsidian Sync.

Sandboxes are non-negotiable for:

• 24/7 personal assistants that control your whole computer like OpenClaw or Hermes;

• non-engineers using tools like Claude Cowork;

• unmonitored remote jobs like Codex;

• or when chasing GPUs and parallel scale.

Is this the only way to add sandboxes to your harness? As we are just getting started, surely not. For another perspective, here is Abhishek Bhardwaj’s talk, explaining how OpenAI runs agent workloads in a cloud of microVM sandboxes instead of containers.

In the next lesson, we will explore the key context-engineering techniques that coding harnesses use: memory, compaction, skills, and LSP servers.

Here is the course roadmap, lesson by lesson (see all in GitHub):

1. Building a Coding Agent From Scratch

2. The Bare-Bones Coding Agent Loop

3. From a Raw Shell to a Sandboxed Coding Agent ← you are here

4. Context Engineering for Coding Agents ← Available next week

5. Agents Catalog, Subagents & Parallel Fan-out

6. Remote Headless Mode & Durability

7. AI Evals Foundations: Benchmarks, Regression and Online

8. AI Evals on Steroids via Replays

But here is what I’m wondering:

Do you run your coding agent raw on your machine, or sandboxed?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

Special thanks to Modal, Opik (by Comet), and Kitaru (by ZenML) for sponsoring this open-source course and keeping it free!

Images & videos

If not otherwise stated, all images are created by the author.

I’m on vacation, so today I will keep it short.

Since February 2026, when I got my $200 Anthropic subscription, I maxed it out only once. It happened when Fable was released, and I started using it the way I used previous models such as Opus. After 4-5 queries through my content and software factories, I was running out of tokens.

Since then, I still use Fable, but only where it’s worth it. Never hitting my weekly token limit. At least not by surprise. I still have those moments where I am aware that if I press Enter, it will cost me tokens. But sometimes it’s worth it.

Anyway, I wanted to share with you 11 context/harness engineering tips that I use in my daily workflows and that work like a charm for me.

1. Progressive disclosure is the foundation everything else sits on

I use references in my AGENTS.md files, LLM wikis, and Obsidian to build hierarchies, so the model walks a table of contents and pulls a file only when something resolves to it. Anthropic frames it as a finite attention budget: every token you add competes with every other one.

OpenAI rebuilt its AGENTS.md into a roughly 100-line table of contents pointing into a linted docs tree, and shipped about a million lines of Codex-written code that way. Quote: “give Codex a map, not a 1,000-page instruction manual”.

2. Plan like crazy, then let a cheaper model execute

Token-wise, planning is cheaper than executing. I use the most powerful model available (currently Fable) to produce a plan, then hand it to a cheaper one. I never say “implement X”. Instead, I build an extremely detailed plan going through the system design, architecture, components, interfaces and data flow. When a smaller model executes that, it’s hard to make a mistake.

I work the plan through 4 dimensions (known knowns, known unknowns, unknown knowns, unknown unknowns) and let the model grill me with questions to surface blind spots.

3. For greenfield work, an LLM wiki is the context you point at

A wiki lets a plan reference 30 documents without loading 30 documents. When I start working on a new project, I always build one first: a light context layer built via files instead of a database. The plan cites the index that describes the documents, not the corpus, so it stays grounded at near-zero cost. I wrote a full piece on this.

4. Send a cheap subagent to read the whole thing

The subagent pays the reading cost. You only receive the summary. When I’m parsing data, I spin up Haiku or Sonnet subagents that hand back only an executive summary to Opus or Fable. A subagent can burn tens of thousands of tokens and return a distilled 1,000-2,000. I clustered my entire Decoding AI archive this way and my main agent’s context barely grew.

5. An ADR and a glossary per feature

For every new feature my planning step outputs a task list, an architecture decision record (ADR), and a glossary update. The ADR caches the why behind each decision; the glossary pins down what each term means. Both act as context for later features. OpenAI recommends the same artifact — “exec-plans checked into the repo” with progress and decision logs.

6. I don’t graph-index my codebase

I don’t build graphs to index my codebase. The code and the docs drift apart almost immediately, and then you burn tokens keeping an artifact in sync with the thing it describes. That violates clean code principles before it’s a token problem.

Note that keeping a glossary and a log of ADRs is different from building a graph out of your code. The glossary and ADRs are orthogonal to the codebase, tracking your domain conventions and design decisions over time. If you change the code, the ADR log doesn’t need to be changed. It just gets a new entry.

Worth being precise, because Cursor is a live counterexample: it diffs its embedding index against file hashes every few minutes and re-uploads only what changed. A strategy that adds a lot of complexity. Meanwhile, Claude Code and Codex just parse the codebase at runtime via glob and grep.

7. Your modules are already the graph

Instead of indexing, I think hard about how to modularize my code, how to shape the interfaces, and how data moves between components. That is your graph, and progressive disclosure parses it really well.

My sanity check: can I paste this module into another project and have it work?

8. Trim the payload you never see

Tools like Claude Code become a Swiss army knife that puts a ton of stuff into the system prompt that you never use. They fill it with references to all their features, such as push notifications, remote triggers, scheduling, etc. The logic is simple: for the harness to work with all those features, they need to be mentioned in the system prompt.

The solution is either to go with a minimalistic approach such as Pi, which ships a bare-bones coding agent you can extend via plugins, or to trim down your Claude Code/Codex harness via a settings.json like this:

Here is how my context window looks now. Hooked into my content & software factory on a 1M-token window, my payload is 21.4k tokens: 4.3k system prompt, 11.6k system tools, 2.7k across 17 skills, and 46 MCP tools costing literally zero because they load on demand.

9. Turn auto-memory off — under one condition

Auto-memory files, like MEMORY.md, reload into the system prompt every session.

I set Claude Code’s auto-memory to false.

Turning memory off is correct when your continuity already comes from committed artifacts you control (AGENTS.md, ADRs, glossaries, LLM wikis).

10. The Caveman plugin

I also use the Caveman plugin to cut token usage. LLMs tend to be overly verbose. This plugin transforms the messages from full sentences into “caveman talk.”

11. The principles that keep every AGENTS.md small

Again, to keep the LLM from being overly verbose and to make it support each instruction with examples, I keep these 3 rules in every AGENTS.md file:

• Always prefer removing instructions over adding more.

• Always use minimum words that still achieve goal — explanations, docs, code.

• New rule in memory (such as AGENTS.md), resources, skills or other files → always support with clear, concise explanation + good and bad examples. Good examples: “a 200-token chunk size”, “sub-100ms latency”. Bad examples: “a powerful architecture”, “a robust pipeline”.

But here is what I’m wondering:

What technique are you using that is not on this list?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

Images

If not otherwise stated, all images are created by the author.

In LangChain’s Terminal-Bench experiment, changing only the harness (with the same model) moved a coding agent from ~30th place into the top 5: the harness, not the model, is what makes a coding agent good.

In the open-source course Building a Coding Agent From Scratch, you’ll build that harness from scratch in Python: Decode, a complete coding agent that grows lesson by lesson from a bare agent loop into a swarm of remote agents running in parallel in the cloud.

Why? You’ll be able to engineer custom harnesses for your own AI products (the skill behind that leaderboard jump), and you’ll understand what Claude Code and Codex actually do under the hood, turning you into a power user.

Lessons:

1. Building a Coding Agent From Scratch

2. The Bare-Bones Coding Agent Loop ← you are here

3. From a Raw Shell to a Sandboxed Coding Agent

4. Context Engineering for Coding Agents

5. Agents Catalog, Subagents & Parallel Fan-out

6. Remote Headless Mode & Durability

7. AI Evals Foundations: Benchmarks, Regression and Online

8. AI Evals on Steroids via Replays

Full open-source course

Lesson 2: The Bare-Bones Coding Agent Loop

When I first tried to implement the code for this course I got greedy. I tried to one-shot the harness in Python: a TUI, an agent, and a handful of tools like bash, read, write, and edit. It worked, until I started adding skills, memory, and compaction. That’s when everything started to fall apart. No “fix this “ prompt could have saved it. That’s when I realized that building a harness for a coding agent is a lot more strategic than just: “Hey, look at OpenCode’s open-source code and build a replica for me.”

So, I took a step back, deleted everything and started researching how Claude Code, OpenCode, Aider and Pi work. Only after I pinned down the whole architecture, with its components, interfaces and data flow I started to build Decode. The coding agent I will teach you how to build in this series.

In this article, lesson 2 from the open-source course Building a Coding Agent From Scratch, we will learn how to build a bare-bones coding agent from scratch. At feature parity with the popular Pi agent harness.

We will build the coding agent loop. See how the loop decomposes a goal into a plan and executes it. Understand which tools are critical for a coding agent, and which ones are nice to have. Build a TUI. See why naively sending messages from the terminal to the agent can corrupt the loop. For the full architecture of Decode, what we will build throughout this series, we recommend reading Lesson 1.

On top of that, we will plug in 3 LLM providers (Modal, OpenRouter, and Gemini) and understand when serverless computing such as Modal beats token-based APIs. Also, to easily debug the agent, we will integrate trace monitoring via Opik.

Example of running Decode (our coding agent) powered by Qwen 3.6 35B:

Let’s start with an end-to-end walkthrough of the coding agent harness.

One Turn, End to End

Our expectations from Decode’s coding agent loop are similar to any other coding agent you’re already used to: Claude Code, Codex, Pi, OpenCode, etc. You open it in a repository or any other folder, give it a goal, and let it loop until it completes the goal.

In our .decode/skills/demo-6-article-kg demo, managed as a skill, we ask Decode to pull data from multiple sources, extract key entities and concepts and render a KG in HTML.

The agent kept looping until we got the graph below. Pretty cool, right?

This is a great example where the agent had to break a goal into multiple steps and execute each one independently until the goal is reached.

More concretely, this is the agent’s whole lifecycle: plan → explore → apply → execute → observe → next task, repeated until the goal is met.

Everything starts with you typing your goal into the TUI and pressing Enter. The request goes into the steering queue and then into a fresh turn, and the loop sends the prompt and tools to the model, a Qwen3.6 served on Modal. It kicks off planning. Via todo_write it records 4 tasks (fetch → extract → render → verify). Via web_fetch Decode fans out in one step and scrapes all the sources. The model reads the scraped output and extracts the KG triplets into a graph.json file via its write tool — the first call that can change your disk, so the run stops and asks. You type y. Ultimately it generates the HTML code that loads the data from the JSON files and beautifully renders it by calling the write tool to create the kg.html file. Finally, it calls todo_write to update the TODO list and mark the goal as done.

Based on our open-source GitHub, here are all the components of the agent loop that transform a simple Pydantic AI agent into the foundations of any coding agent you are using out there:

src/decode/
├── agent/          # loop.py (the agent loop), factory.py
├── harness/        # runner.py (single-flight turns), queue.py (steering/follow-up)
├── tools/          # registry.py, files.py (read/write/edit/glob/grep), bash.py, askuser.py
├── context/        # session_log.py (append-only JSONL)
├── observability/  # tracing.py (Opik via OTLP)
└── tui/            # app.py (prompt_toolkit input), render.py (Rich output)

In this course, we will mostly follow Claude Code/OpenCode’s design, where we have a larger set of tools, memory and agent features built into the harness. Mostly for educational reasons, to explore all the options out there. But there is also a beautiful, minimalist approach to this, based on the philosophy of Mario Zechner, creator of the Pi harness: strip the agent loop back to its bare bones, keeping only the essential logic, to avoid adding noise into the context window and keep everything observable and in control. If you want more features, you add them explicitly as plugins, instead of getting new “space ship” cool-but-useless features with every version. Thus, to understand both perspectives, we will keep drawing parallels to Pi’s architecture.

The Agent Loop

In Lesson 1, we said that the agent loop is the good old ReAct pattern. The model reasons, picks a tool, the harness executes it, and the observation feeds the next step — an agent is just an LLM using tools on environmental feedback in a loop.

ReAct in the abstract is reason → act → observe. Give it a set of tools it can operate on a codebase and voilà, it becomes a coding agent.

Plan: todo_write the task list before touching anything.
Explore: read, glob, grep the codebase.
Apply: write and edit, stopping for your verdict.
Execute: bash the tests, the linter.
Observe: a failure is the next observation, and the repair is another Apply → Execute pass on the same task. Tick it off, re-enter.

In Claude Code’s leaked source the core loop is ~150 lines. To emphasize the harness around the agent, we will use Pydantic AI for the “core agent” and focus on everything around it (from build_agent in src/decode/agent/factory.py):

def build_agent(*, model: str | None = None) -> Agent[AgentDeps, str | DeferredToolRequests]:
    built_model = _build_model(model=model)
    agent = Agent(
        built_model,
        deps_type=AgentDeps,                       # cwd, task store, event emitter — the harness state
        output_type=[str, DeferredToolRequests],   # a step ends in text — or pauses on approvals
        output_retries=3,                          # empty / thinking-only turns get another attempt
        tool_retries=5,                            # only resets on success — the plan-survival budget
    )
    register_tools(agent)                          # the coding tool set
    _register_instructions(agent)                  # ONE merged system prompt

    return agent

The Agent owns one step: one model request, the tool executions it triggers as “actions”, and the results it gets back as “observations”, ending in either text or DeferredToolRequests. The agent loop is operated by the AgentTurnHandler (src/decode/agent/loop.py) class chaining steps together through agent.iter.

AgentTurnHandler.__call__ is an async generator that streams tokens in real time via Python generators. It runs a while True loop with two boundaries: yield Boundary.MODEL_REQUEST and yield Boundary.WOULD_STOP. Via Boundary.MODEL_REQUEST, we can inject (steer) new messages into the loop before it finishes reaching its goals.

class AgentTurnHandler:

    ...                 # ONE instance per session; owns the turn + history

    async def __call__(self, ctx: TurnContext) -> AsyncGenerator[Boundary, list[str]]:
        while True:
            steering = yield Boundary.MODEL_REQUEST
            if pending_results is not None:
                self._append_steering(steering)
                output = await self._run_turn(ctx, deferred_results=pending_results)
                pending_results = None
            else:
                output = await self._run_turn(
                    ctx, prompt=self._compose_prompt(next_prompt, steering)
                )
            if isinstance(output, DeferredToolRequests):
                pending_results = await self._resolve_deferred(ctx, output)
                continue
            self._persist_turn()
            follow_ups = yield Boundary.WOULD_STOP
            if not follow_ups: return
            next_prompt = "\n".join(follow_ups)

The whole action happens inside the self._run_turn method, which makes the actual model call, and based on the results from the model, via the run object, it either emits answers from the model or executes a tool request.

In more detail, this is what happens:

1. A fresh step passes prompt, a resumed step passes deferred_tool_results, never both.

2. The async for streams each node to the TUI as an event, so you watch tokens arrive live.

3. The finally carries run.all_messages() forward, so a step that dies keeps everything it accomplished in memory.

class AgentTurnHandler:

    ...                         # ...same class as above

    async def _run_turn(self, ctx, *, prompt=None, deferred_results=None) -> str | DeferredToolRequests:
        async with self._agent.iter(
            prompt,
            deps=self._deps,
            message_history=self.message_history,
            deferred_tool_results=deferred_results,
        ) as run:
            try:
                async for node in run:
                    if Agent.is_model_request_node(node):
                        await self._stream_model_node(ctx, node, run)
                    elif Agent.is_call_tools_node(node):
                        await self._stream_tool_node(ctx, node, run)
            finally:
                self.message_history = run.all_messages()
                self._last_input_tokens = _leg_input_tokens(self.message_history)
        return run.result.output

💡 The loop has no max-steps knob, based on Pi’s principles: “the loop just loops until the agent says it’s done.” A cap is a guess about how many steps a task needs, and the model already signals completion by returning text instead of a tool call. The signal you should be careful about is the context window.

The first thing every step does is call a model, and picking that model should always be a one-line config choice.

The LLM Providers

It’s an architectural mistake to couple your design to a single provider. The loop shouldn’t know whether it talks to Gemini, OpenRouter, Modal, or whatever comes next.

All provider knowledge lives in _build_model() in src/decode/agent/factory.py. You pick the provider by setting the LLM_PROVIDER environment variable, which is picked by your Pydantic settings object from src/decode/config/settings.py.

def _build_model(*, model: str | None = None) -> Model:
    provider = settings.llm_provider                     # explicit LLM_PROVIDER env var
    if provider == "gemini":
        return GoogleModel(
            model or settings.gemini_model,
            provider=GoogleProvider(api_key=_provider_api_key("gemini"))
        )
    if provider == "openrouter":
        return OpenAIChatModel(
            model or settings.openrouter_model,
            provider=OpenRouterProvider(api_key=_provider_api_key("openrouter"))
        )
    if provider == "modal":
        client = AsyncOpenAI(
            base_url=f"{settings.modal_endpoint_url}/v1",
            api_key=token_secret,
            default_headers={"Modal-Key": token_id, "Modal-Secret": token_secret}
        )
        return OpenAIChatModel(
            model or settings.modal_endpoint_model,
            provider=OpenAIProvider(openai_client=client)
        )
        
    raise ValueError(f"unsupported llm_provider: {provider!r}")

So from all the options out there (we know there are tons), why these 3? And no OpenAI or Anthropic. Because between them they cover the three tiers of the build vs. buy decision: Gemini is proprietary (you buy the model), OpenRouter is open weights bought as a service (you buy the serving), and Modal is open weights you serve yourself.

We choose Modal as our default option for 3 reasons. You pay for GPU compute, not tokens: agentic bursts fit GPU-time pricing. Serverless open-source models supported by Modal that we deployed are Qwen3.6 35B, Kimi K3, and GLM-5.2 via Modal Endpoints. If you want to use a fine-tuned model, you can easily swap it in with 0 changes on the harness side. Because it’s serverless, it scales to zero: you pay only when the agent is working.

This combination makes it ideal for intermittent periods of extremely high compute demand. For example, if you want to process 1000 documents, instead of paying for each token, which is what’s typical of all the APIs out there, you just pay for the compute time. To make testing Decode extremely affordable, for most of our tests we used Qwen3.6 35B, which runs on a single H200 GPU at $4.54 / h.

But going back to our 1000 documents example, if these 1000 documents translate to 1000 * 30000 tokens, that's 30M input tokens, plus ~500 output tokens per document. If we wanted to process all these tokens with Sonnet, that would translate to: 30M / 1M x 3 (input) + 500k / 1M x 15 (output) ~= $97. And prompt caching doesn't save us here — every document is a different prefix, so there's nothing to reuse across calls. Considering we could process this data with Modal at ~3000 tokens/second batched, that's under 3 hours of GPU time, which would have cost only ~$13. Modal benchmarked the same tradeoff and found that giving up interactivity, from ~200ms to ~4s end-to-end, provided an x8 throughput increase on identical hardware. If we do this multiple times per day, it quickly adds up. This is just napkin math, but you get the idea.

Now, on the other side of the spectrum, let's assume Decode sits idle while it waits on a “y” confirmation the agent asked for. If that happens overnight, you can easily add 10 hours of idle time to your GPU billing, an extra ~$45.

Alongside the pay-per-token model, which quickly becomes inefficient at high volumes, another dimension to consider is serverless vs. reserved GPU time. When you reserve GPUs, you have to reserve your peak. If your peak needs 40 GPUs on Friday afternoon and 2 on Sunday night, you sign for 40 — and you keep paying for 40 all weekend, all month, all year. With serverless, the meter follows the demand curve, so the total cost is just the rate R_s times whatever you actually used at each moment. The reservation instead charges you rate R_r times the peak, times the entire length of the contract T.

In other words, if reserving is 5x cheaper than serverless, but your peak is 5x higher than your average, then you break even. And the further your peak climbs above that price advantage, the better off you are paying for serverless. For the full math, see this blog post.

OpenRouter is the most popular gateway to every hosted model: closed source (Claude, OpenAI, Gemini) or open-source (Kimi K3, Z GLM-5.2, Gemma 4, etc.) behind one key. Thanks to their huge aggregated pool of models, they also have some free defaults, such as qwen/qwen3-coder:free, which we used in this course. Having them integrated into your app is a powerful way to easily experiment with different models without worrying about the underlying infrastructure.

We picked Gemini mostly due to their generous free tier, to provide you a third option to run Decode without any cost. Their models are rarely the best, but relative to other API-based models, especially OpenAI & Anthropic, they’re cheap and good enough to get most jobs done.

We recommend starting with Modal Endpoints, powered by Qwen3.6 35B by default, that spins up in a couple of minutes a configurable SGLangEndpoint. SGLang is the right default here because it and vLLM land in roughly the same place on throughput, but SGLang boots in ~1 minute against vLLM’s ~5 for small models, which is everything when your endpoint scales to zero. Modal’s $30 in free credits each month covers ~6.5 hours of H200 time, enough to run the whole course at no cost. And while Qwen3.6 35B is small next to GLM-5.2 740B or even GPT OSS 120B, it’s well tested with Decode and works well enough for our examples.

After you’ve completed the basic setup on Modal, you can start a new endpoint just by running:

modal endpoint create --model Qwen/Qwen3.6-35B-A3B-FP8 --env main

While trying it out, to avoid the cold start problem, we recommend setting the autoscaling limits to a minimum of 1 container:

🧑‍💻 Find the full setup docs in our GitHub repo.

But between Modal and OpenRouter, you can easily swap between hundreds of models. You can try out whatever you’re interested in. The only requirement is that it supports tool calling.

The System Prompt

The entire base system prompt is one paragraph:

You are Decode, a terminal coding agent that helps a developer in their working
directory. You are concise and precise: answer directly, prefer running the work
over describing it, and never invent file contents or command output you have not
seen. When you do not have a tool for something yet, say so plainly rather than
pretending.

The team behind Pi started this minimalist mentality, where you keep your system prompts, tools, and features to a minimum, to avoid filling your context window with noise that confuses the model more than it helps. This also makes maintaining and understanding the prompts, for us as humans, a lot easier. In the past months, OpenAI and Anthropic followed, trimming their system prompts by up to 70%, as more and more functionality around coding gets baked directly into the model’s weights.

Still, this paragraph is only the baseline. Pydantic AI builds a schema from every tool’s signature, which is added to the system prompt. Thus, each tool comes with a cost, meaning that the more tools you use, the more context window you consume, and the model becomes more confused about which one to pick. This becomes even more problematic when we start adding memory and skills into the mix, as we will see in future lessons.

Whichever model answers, it can only touch your repo through the tools.

The Core Tools

In reality, the tools are 90% of why this is a coding agent and not any other kind of AI agent. The agent needs to read files, execute programs, make HTTP requests. Pi ships exactly read, write, edit, and bash — under 1,000 tokens of prompt plus definitions, still respectable — top 10 on Terminal-Bench 2.0.

read   path, offset?, limit?   → file contents; first 2,000 lines by default
write  path, content           → create or overwrite; makes parent dirs
edit   path, oldText, newText  → exact-match surgical replace (whitespace included)
bash   command, timeout?       → run it in the cwd; returns stdout + stderr

On top of these 4 core tools, we have more specialized I/O tools such as glob and grep. Then there’s todo_write, used to decompose the goal into a plan of TODOs, web_fetch to fetch web pages, and ask_user to interact with the user.

💡 Pi doesn’t have a todo_write tool because it considers that the best way to store your plan is directly in a Markdown file, not in an in-memory TODO list the agent has to keep track of — a PLAN.md on disk beats an in-memory list.

Every snippet below opens with ctx: RunContext[AgentDeps]. RunContext is Pydantic AI’s per-run context object: the framework injects it as every tool’s first parameter and hides it from the model’s tool schema. The model calls edit(path, old_string, new_string) and never knows ctx exists.

AgentDeps is in our harness. We declare it once as deps_type=AgentDeps in build_agent, pass it into agent.iter(deps=...), and Pydantic AI hands it to every tool call. RunContext[AgentDeps] reads as “this run’s context, whose .deps is an AgentDeps.” It is the harness state, arriving by injection (from src/decode/agent/deps.py):

@dataclass(slots=True)
class AgentDeps:                                   # every tool's ctx.deps — the harness, injected
    cwd: Path                                      # the jail: file tools + bash resolve paths here
    emit: EventSink                                # push an event → the TUI renders it
    gate: PermissionGate                           # the allow / ask / deny policy (Section 6)
    resolve_permission: PermissionResolver         # a gate "ask" → the human's verdict, awaited
    resolve_user_question: UserQuestionResolver    # ask_user's answer, on the SAME channel
    task_store: list[Task] = field(default_factory=list)      # what todo_write rewrites in place

emit, resolve_permission, and resolve_user_question are used for dependency injection, so tools reach the terminal without importing the TUI, and they make this code easily testable by swapping in a different AgentDeps.

To keep things short and sweet, I will skip how read and write work and go straight to edit (see them at src/decode/tools/files.py). edit is a find-and-replace with one rule: load the file’s text, find old_string inside it, dump new_string into that exact spot, write the file back. But the devil is in the details. To make this replace work, it first needs to normalize away the invisible formatting the model was never shown — a file’s line-ending convention, a leading byte-order mark — so a match never fails on characters the model had no way to guess. Then it looks for old_string, and only when that misses does it retry with whitespace collapsed on both sides. Either way, the match must be unique: 0 hits returns not found, 2+ returns ambiguous, N matches, and both end up with ModelRetry:

def edit(ctx: RunContext[AgentDeps], path: str, old_string: str, new_string: str) -> str:
    if needs_approval(ctx):
        raise ApprovalRequired
    target = _resolve_in_cwd(ctx.deps.cwd, path)
    raw = target.read_bytes().decode("utf-8")
    final = _apply_edit(raw, old_string, new_string)
    _atomic_write_bytes(target, final.encode())

    return f"Edited {path!r} (replaced 1 occurrence)."

Within todo_write, the model sends the full desired list on every call and the tool overwrites the whole list — which is exactly what stops the plan from drifting out of the context window. The overwrite is in place ([:] = tasks, not a rebind) so the loop and the TUI keep pointing at the same list object, and every item is validated by the Task Pydantic model on the way in. It then emits one TaskListUpdated event carrying pre-rendered checklist lines: the tool maps pending / in_progress / completed to [ ] / [~] / [x] itself, so the renderer never has to learn the status vocabulary. The store is in-memory and per-run — nothing about the task list outlives the process (from src/decode/tools/tasks.py):

def todo_write(ctx: RunContext[AgentDeps], tasks: list[Task]) -> str:
    if needs_approval(ctx):
        raise ApprovalRequired
    ctx.deps.task_store[:] = tasks
    ctx.deps.emit(events.TaskListUpdated(tasks=_checklist_lines(tasks)))

    return f"Updated task list ({len(tasks)} task(s))."

That emit line is the harness’s entire UI protocol. events is the decode.entities.events module of frozen dataclasses (TurnStarted, ToolCallStarted, ToolResult, TaskListUpdated, ...) joined into one union known by the renderer.

Event = (
    TurnStarted
    | TurnFinished
    | AssistantTextDelta
    | ThinkingDelta
    | ToolCallStarted
    | ToolResult
    | PermissionRequested
    | AskUserRequested
    | TaskListUpdated
    | ContextCompacted
    | ContextMicrocompacted
    | AgentError
)

ctx.deps.emit is a synchronous event-based sink, injected into the harness to communicate with outer layers, in this case the TUI. This is clean architecture 101. The tool describes what happened as a typed value, and the TUI decides how to draw it.

web_fetch makes an async httpx GET request. What comes back is filtered twice before the model sees it — non-text content types are refused outright, and the decoded body is hard-capped at 2 MB, so an enormous page can never eat your entire context window. HTML is then mapped to Markdown keeping mostly the body and other metadata such as the title. Every failure path — non-2xx, timeout, connection error, wrong content type, HTML nested too deep to parse — throws a ModelRetry error, so a bad fetch is an observation the model has to handle explicitly and not ignore (from src/decode/tools/web.py):

async def web_fetch(ctx: RunContext[AgentDeps], url: str) -> str:
    if needs_approval(ctx):
        raise ApprovalRequired
    _validate_url(url)
    response = await _get(url)
    body = _render_body(response, _decode_body(response))

    return f"HTTP {response.status_code} {response.url}\n\n{body}"

ask_user lets the agent request input from the user. It emits AskUserRequested so the TUI renders the question, then awaits ctx.deps.resolve_user_question. Two things can go wrong: the agent loop can run in headless mode, where there is no interactive user at all, or the question can be dismissed (turn aborted, REPL shutting down). Both become ModelRetry text telling the model plainly to proceed without an answer (from src/decode/tools/askuser.py):

async def ask_user(ctx: RunContext[AgentDeps], question: str) -> str:
    ctx.deps.emit(events.AskUserRequested(question=question))
    try:
        return await ctx.deps.resolve_user_question(question)
    except NoInteractiveUserError as exc:
        raise ModelRetry(str(exc)) from exc

bash is the most powerful. Via bash the agent can execute any shell command such as ls, cd, mkdir, rm, cp, mv, and echo. Plus, piping and everything else you can run in a terminal. Which means that via bash the agent can run Python, TypeScript, Rust, or any other language. Via bash it can manipulate all the CLIs you give it access to.

Via bash it gets that magical feeling of being able to interact with the operating system and run any command.

But this also makes it the most dangerous one. rm -rf ~/ is always only a few tokens away. That’s why running your agent in a sandbox environment becomes so important. In a future lesson, we will get into the executor and how to run commands in local Docker and remote Modal containers.

Bash plus code execution is a big step toward giving the model a computer (from src/decode/tools/bash.py):

async def bash(ctx: RunContext[AgentDeps], command: str, timeout: float | None = None) -> str:
    if needs_approval(ctx):
        raise ApprovalRequired
    timeout_s = _resolve_timeout(timeout)
    result = await _get_executor().run(command, cwd=ctx.deps.cwd, timeout_s=timeout_s)

    return _render(result, timeout_s=timeout_s)

As we’ve seen at the beginning, the lifecycle of the agent is based on running these tools in order: todo_write plans, read / glob / grep explore, write / edit apply, bash executes (bash‘s exit code is the observation that starts the next pass).

When running .decode/skills/demo-6-article-kg that asks the agent to scrape 3 web pages and build a knowledge graph, the agent creates a 4-step plan, keeps track of it via todo_write, fetches the data via web_fetch, extracts the KG triplets directly, then via write saves them to graph.json. Next, it writes the HTML and CSS code that displays the KG triplets into kg.html. Ultimately, via bash it checks that the HTML code renders correctly. If not, the model edits the code and retries.

Adding Tracing to Understand What Is Going On

We’ve seen above how the agent calls the todo_write tool to bake the plan into its state. The trace is logged via Opik. Let’s see how easy it is to plug it into your own code.

Debugging your agent via the terminal is terrible. Not only is reading the full traces hard, sometimes it’s impossible, as you just don’t have access to all the spans, plus other essential metadata such as configs, token counts, tool inputs and outputs — and the list goes on. That’s why you need to plug in an observability tool like Opik from day 0 to get visibility into your agent’s behavior.

As we use Pydantic AI, it has native support for tracing via Logfire, the observability tool made by Pydantic. Logfire, like most observability tools, supports OpenTelemetry (OTLP) out of the box, which is the standard wire protocol for tracing. Thus, hooking up Opik is as easy as defining the OTLPSpanExporter to point to Opik’s managed servers via the endpoint parameter, plus authentication headers. As Opik is open-source, this would work exactly the same if you were hosting it yourself (from src/decode/observability/tracing.py).

def init_tracing() -> bool:
    key = settings.opik_api_key.get_secret_value()
    if not key:
        return False

    exporter = OTLPSpanExporter(
        endpoint=f"{base}/v1/traces",
        headers={"Authorization": key, "Comet-Workspace": settings.opik_workspace,
                 "projectName": settings.opik_project_name})
    logfire.configure(send_to_logfire=False, console=False,
        additional_span_processors=[BatchSpanProcessor(CostAnnotatingExporter(exporter))])
    logfire.instrument_pydantic_ai()

    return True

You can find the full setup on how to get started with Opik in our GitHub repository. Opik is optional to get through the course, but note that it has a generous freemium plan with 25k spans/month free. You can run the whole course for free.

In the GIF below, you can see an end-to-end trace of how the agent executed the .decode/skills/demo-6-article-kg skill that builds the KG from 3 articles. It consists of 47 spans, covering 6 steps, 15 model calls, and 18 tool executions over ~6m.

With the following token usage (no cost per token as we ran it via Modal):

cache_read.input_tokens: 631424
completion_tokens: 19781
details.cache_read_tokens: 631424
details.reasoning_tokens: 2912
prompt_tokens: 655624
total_tokens: 675405

The TUI and the Queues

You can build a TUI in 2 ways. Full-screen takes over the terminal’s viewport as a grid of cells (Amp, OpenCode): total layout control, but you lose scrollback, search and have to implement a bunch of features from scratch. Append-to-scrollback writes like an ordinary CLI (Claude Code, Codex, Pi). Decode takes the second approach.

As Python packages, we use prompt_toolkit to read the input and Rich to render output in append-style. Below you can see the loop that takes events from the harness and renders them on the TUI (from src/decode/tui/app.py).

with patch_stdout(raw=True):
    while True:
        submitted = await session.prompt_async(_PROMPT)   # one pinned input line
        intent, text = _interpret(submitted)              # Enter / Alt+Enter / Esc
        if decisions.pending:                             # mid-turn question? answer it
            decisions.resolve(text)
            continue
        await runner.submit(text, intent)

But what happens when you type a new command mid-turn, while the agent is still running your previous one? Injecting it instantly would corrupt the current tool call. Thus, you need to buffer your input on arrival and inject it only at boundaries, such as after a tool finishes running or a whole turn ends.

The input can arrive in 3 modes.

• Steering (plain Enter): the Runner executes it within a model turn, between tool calls.

• Follow-up (Alt+Enter): the Runner holds it until the turn stops, then runs it as one more step.

• Cooperative abort (Esc): sets a flag to stop the agent turn at the next boundary. Next, it clears both queues. This avoids corrupting the history.

As seen within the InteractionQueues class, for steering and follow-ups we use two different in-memory queues (from src/decode/harness/queue.py).

@dataclass(slots=True)
class InteractionQueues:
    steering: asyncio.Queue[str] = field(default_factory=asyncio.Queue)
    follow_up: asyncio.Queue[str] = field(default_factory=asyncio.Queue)
    def drain_steering(self) -> list[str]:  return _drain(self.steering)
    def drain_follow_up(self) -> list[str]: return _drain(self.follow_up)
    def clear(self) -> None:                # on abort: drop everything pending
        _drain(self.steering)
        _drain(self.follow_up)

This loop belongs to the Runner, which is the facade between the TUI and the agent. Its job is to keep the agent alive as long as the TUI session is running. It also takes care of draining the queues at the right moments and sending them to the agent (from src/decode/harness/runner.py).

class Runner:

    ...                 # single-flight: one turn in flight, ever

    async def _run_turn(self, turn_id: int, prompt: str) -> None:
        agen = self._turn_handler(ctx)
        boundary = await agen.asend(None)
        while True:
            if self._abort:
                aborted = True
                break                                 # Esc: stop at the boundary, keep history
            if boundary is Boundary.MODEL_REQUEST:
                sent = self.queues.drain_steering()   # steering: before every model call
            elif boundary is Boundary.WOULD_STOP:
                sent = self.queues.drain_follow_up()  # follow-up: only when the turn would end
            try:
                boundary = await agen.asend(sent)     # resume the loop; get the next boundary
            except StopAsyncIteration:
                break

This is the approach implemented in Pi. Another option, implemented in OpenCode, is to keep track of the session in your database, which means you can use your database as the queue as well. But that also means that you need to complicate your solution with a database. Which raises the question, how do we keep track of sessions?

The Session Log

In the interactive mode the conversation is the working state. To keep track of sessions, we create a session log — an append-only JSONL file per session under .decode/sessions/. This simple approach kills the need for a database entirely. This is how Claude Code implements their session management.

.decode/sessions/
├── 20260620T132411Z_d710a5b9-28af-418b-9a15-936942a8db12.jsonl
├── 20260620T132419Z_058cddc8-1d0f-406d-a754-098703c07cb5.jsonl
├── 20260622T110408Z_7114fe38-d294-4c78-bbc3-c454be57082b.jsonl
├── 20260622T111609Z_4bbf140e-3495-4d69-9a88-c7014fd77742.jsonl
├── 20260625T152011Z_7594e9b0-6c62-421a-ad74-9918c4d0291e.jsonl
└── ...

Every completed turn appends one messages line to the log. An aborted or crashed turn persists what it finished (from src/decode/context/session_log.py).

def append_turn(self, new_messages: list[ModelMessage]) -> None:
    if not new_messages:
        return

    payload = json.loads(ModelMessagesTypeAdapter.dump_json(new_messages))
    entry = {"type": "messages", "messages": payload}
    with self.path.open("a", encoding="utf-8") as handle:
        handle.write(json.dumps(entry) + "\n")

Based on the knowledge-graph example, here is how the log entries look:

{"type": "session", "version": 1, "session_id": "48900d84…", "cwd": "…/building-a-coding-agent-from-scratch-course", "created_at": "2026-07-16T19:32:01Z"}
{"type": "messages", "messages": [
    {"role": "user", "content": "The hardcore one: turn three live articles into a knowledge graph…"},
    {"role": "tool", "name": "web_fetch", "content": "HTTP 200 …/keep-knowledge-graph-clean\n\nHow to Keep a Knowledge Graph Clean…"},
    {"role": "tool", "name": "bash", "content": "Exit code: 0.\n\nstdout:\nNode count: 28 (should be 20-35)\nEdge count: 29…"},
    {"role": "assistant", "content": "## Knowledge Graph Visualization — Complete…"}]}

To resume the state, you can run decode --resume <session_id> to kick off a new TUI from an existing session log.

Next Steps

In this lesson, we’ve implemented a bare-bones version of the coding agent, adding the right tools, a TUI, and a way to manage sessions. More or less, at this point, we are at feature parity with Pi.

But as a course on harness engineering, we want to push it further. Any agent needs a headless mode (along with running it from the terminal), essential context engineering features such as memory, compaction, skills and guardrails such as the permission layer and sandboxes. That’s essentially harness engineering. So far we’ve built the agent loop. Now the real harness engineering begins.

In the next lesson, we’ll build the sandbox layer locally via Docker and remotely via Modal.

Here is the course roadmap, lesson by lesson (see all in GitHub):

1. Building a Coding Agent From Scratch

2. The Bare-Bones Coding Agent Loop ← You are here

3. From a Raw Shell to a Sandboxed Coding Agent

4. Context Engineering for Coding Agents

5. Agents Catalog, Subagents & Parallel Fan-out

6. Remote Headless Mode & Durability

7. AI Evals Foundations: Benchmarks, Regression and Online

8. AI Evals on Steroids via Replays

Or the video lecture supporting the first two lessons:

But here is what I’m wondering:

From your perspective, as we’ve been using Claude Code, Codex, etc. for general productivity stuff, what makes a general-purpose agent a coding agent?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

Special thanks to Modal, Opik (by Comet), and Kitaru (by ZenML) for sponsoring this open-source course and keeping it free!

Images & Videos

If not otherwise stated, all images are created by the author.

In LangChain’s Terminal-Bench experiment, changing only the harness (with the same model) moved a coding agent from ~30th place into the top 5: the harness, not the model, is what makes a coding agent good.

In the open-source course Building a Coding Agent From Scratch, you’ll build that harness from scratch in Python: Decode, a complete coding agent that grows lesson by lesson from a bare agent loop into a swarm of remote agents running in parallel in the cloud.

Why? You’ll be able to engineer custom harnesses for your own AI products (the skill behind that leaderboard jump), and you’ll understand what Claude Code and Codex actually do under the hood, turning you into a power user.

Lessons:

1. Building a Coding Agent From Scratch ← you are here

2. The Bare-Bones Coding Agent Loop

3. From a Raw Shell to a Sandboxed Coding Agent

4. Context Engineering for Coding Agents

5. Agents Catalog, Subagents & Parallel Fan-out

6. Remote Headless Mode & Durability

7. AI Evals Foundations: Benchmarks, Regression and Online

8. AI Evals on Steroids via Replays

Full open-source course

Lesson 1: Building a Coding Agent From Scratch

In a test run by LangChain on Terminal-Bench, changing only the harness (same model throughout) moved a coding agent from roughly 30th place into the top 5. The model isn’t what makes a coding agent good. The harness is. So that’s what we’ll build from scratch. A coding agent harness.

When I first started digging into coding agents, I couldn’t tell where the agent ended and the harness began. I wanted past that black box, so in the past months I’ve researched how the most popular coding agent tools work under the hood: Claude Code (via its leaked source), OpenCode, Pi, and Aider.

That harness is the only layer you can actually engineer. And that’s why I think building your own coding agent harness from scratch is the best way to understand both how to better use coding agents to become a power user and how to build your own custom harnesses for AI agents.

During my research, I’ve built my own replica: “Decode: The Coding Agent”. Now, I want to show you how to build your own.

I don’t mean a simple LLM with some tools in a loop, which is just the good old ReAct pattern. I mean building the whole coding harness from scratch. Build a coding agent once, and you’re equipped to build a custom agent for any use case.

The truth is that to add custom business logic to existing open-source harnesses such as Pi, DeepAgents by LangChain, or Pydantic AI Harness is not that hardcore. But to know what to add based on the internals of these harnesses is extremely important. That’s fundamentals. The secret sauce that still makes AI Engineers valuable.

In this 8-lesson series, you’ll build a complete coding agent called Decode, from scratch, in Python. It runs as a CLI via decode for interactive use, or as a remote headless agent that lets you run a swarm of agents in parallel on a cloud platform for automation such as picking up tasks from the backlog or PR reviews.

You’ll learn how to implement all of that in 3 escalating phases. First, build: the agent loop via Pydantic AI, powered by open-source models on Modal or OpenRouter.

Then building the harness around the loop: durable execution with Kitaru, context engineering, memory, the permission layer, sandboxing in Docker and Modal, and the agent catalog of planners, builders, explorers, and subagents. All the good stuff.

Finally, the boring stuff that makes you a senior AI engineer: a custom benchmark similar to Terminal-Bench, AI Evals for regressions, observability via Opik, and deployment to GCP running a swarm of agents in parallel.

Before getting into the implementation, I want to kick off this lesson with the most fun part: the system design of the coding agent.

Let’s trace the architecture end to end, following one request through every component.

The High-Level System Design

You pass a coding agent a request — “fix the bugs that make the unit tests fail” — and it loops: explore the codebase, run the tests, read the feedback, edit the code, repeat. The input is your request, while the repository is the state of the codebase. The output is tested changes done to the repo. Everything in between is the harness.

We are pretty familiar with this process. But how does it work?

At the center sits the headless harness: no interface of its own. It exposes a module that interfaces connect to. At the core, it runs the agent loop every harness shares: the LLM picks an action (a tool call — run the failing test suite), the tool executes and returns an observation (2 failing tests, expected vs actual), and the loop feeds it back. Everything reads from and writes to the context window (the state of the LLM).

To highlight that the harness is independent from the LLM, we wanted to implement multiple LLM providers. The loop is powered by an LLM provider module that to support open-source models we implemented it with Modal (for self-hosting) and OpenRouter (for APIs). Also, we added support for Gemini for their free tier. Modal is the most interesting as it allows us to either host standard open-source models or easily host our own fine-tuned models on top of their serverless infrastructure.

That cycle lets the agent correct itself: implement the code, run or compile it, read the error, fix it — or catch the mistake before anything runs, with a Language Server Protocol (LSP) server flagging broken syntax the moment an edit happens. The tighter these feedback loops, the faster the agent converges on working code. The LSP server (we used ty - made by Astral’s - the guys behind uv and ruff) is the cheapest way to get feedback on code changes.

Along with the LLM providers and LSP server, the harness contains 4 more essential modules: Memory, Skills, Sandbox, and Permissions. Essential for context engineering and security.

The context window of the LLM is a budget. The conversation accumulates in the window until it stops fitting or performance degrades. Remember that too much context confuses the model, known as context decay. Every observation the loop feeds back spends from the same finite budget. Hence, the harness needs to implement compaction techniques such as summarization, truncation, or just clearing the window.

How do we interact with the headless harness? It has two core interfaces: interactive mode, a terminal user interface (TUI) wired to one live session, and remote mode, where an agent runtime (Kitaru) runs N headless harnesses in parallel on a server.

On top of everything, we have the AI evals & observability layer, powered by Opik, that records every model call and tool call, and turns a bad prompt tweak into a failing regression score before your users feel it in production.

Here is how all the components fit together:

You type the request into the TUI. The input goes into the steering queue through the priority gate into the session’s context window. The loop sends the window (with the available tools) to the model through the LLM provider. The model answers with an action — grep for the failing assertion, then edit stats.py. Permissions checks if it can execute the tool, the tool executes inside the Sandbox, and the observation is added back into the context window. The loop repeats until the model stops calling tools. During the loop execution, events are streaming back to your terminal in real-time via an async generator (or SSE for remote streaming events).

Next, let’s zoom into the agent loop, where the agent ends and the harness begins, the boundary that defines this entire series.

The Headless Harness & The Agent Loop

The agent itself is a Pydantic AI agent. A ~20 line definition that composes the model, the tools, the output type and iterates until the model stops calling them. The snippet below is the distilled version of Decode’s loop:

from dataclasses import dataclass
from pydantic_ai import Agent, DeferredToolRequests

@dataclass
class AgentDeps:                # the harness state injected into every tool call
    cwd: Path                   # the repo the tools operate on
    emit: EventSink             # streams events to whatever interface is attached
    gate: PermissionGate        # allow / ask / deny, per tool call
    resolve_permission: PermissionResolver  # asks the human when the gate says "ask"

agent = Agent(
    build_model(settings.llm_provider),       # gemini | openrouter | modal
    deps_type=AgentDeps,
    output_type=[str, DeferredToolRequests],   # a turn ends as a final answer (str),
)                                              # or as tool calls paused for approval
register_tools(agent)                          # read, edit, bash, grep, ...

async with agent.iter(prompt, message_history=history) as run:
    async for node in run:                     # model request → tool calls → repeat
        stream_events(node)                    # everything else is the harness

Through the AgentDeps dataclass we inject harness related components into the loop. The tools are never aware of the interface: TUI or headless. This is what I like to call “a practical clean architecture”, where you are obsessive about separating the serving and infrastructure layers from your core business logic via interfaces and composition.

output_type names the only 2 ways a turn can end: a final answer, or tool calls suspended mid-turn waiting for human input.

We will go into all the details about the agent loop, explaining how we hook multiple closed-source/open-source models, the tools and other deps into the loop, in the next article.

These ~20 lines are the entire tool-calling LLM agent. The thing people call “the agent” ends here. Everything we build on top of it across 8 lessons is the coding harness.

Pydantic AI is a thin layer over the model APIs: typed tool definitions, streaming, structured outputs, with no heavy framework between us and our own logic. What makes this a coding agent is the tools: read and edit files, run bash, search the codebase, manage a task list, fetch the web, ask the user a question, and spawn subagents. Plus, the LLM specializes in coding. Anthropic is well known for continually optimizing their models for coding tasks. That’s why with every release their models get worse and worse at writing, while better and better at coding.

Still, almost everyone already uses coding agents such as Claude Code or Codex for completely different use cases (including myself): productivity, content creation, finance, sales, marketing. The line between a coding agent and a general-purpose agent is blurry, as any agent needs to natively interact with your computer. How would a harness specialized in writing look? Other than making the LLM sound better. Which is not the harness!

On top of the agent loop, we have 6 modules + compaction that transform the agent loop into a headless harness.

The Six Modules + Compaction

LLM Providers — Decode ships with 3: Modal to serve any type of open-source models (pre-configured or your own fine-tuned version), deployed as Modal endpoints — serverless GPU web endpoints that put the model behind a URL the harness calls like any hosted API, OpenRouter to reach any hosted model behind one API, and Gemini for Google’s free tier for following along. Also, by implementing three providers we can showcase how to swap the model only with a config change, not a rewrite. The loop should never know which model is powering the harness. Clean architecture, remember?

Sandbox — an agent that runs bash can break things. We wrap the bash and file-I/O tools to execute inside a sandbox (Docker locally, Modal Sandboxes remotely) instead of on your machine.

Permissions — the permission layer is the most important guardrail: it asks you before running every action, or only the risky ones, depending on the mode. For example, it pauses before a destructive bash command like rm -rf and waits for your approval. This is similar to Claude Code’s default, edit, and auto modes.

Memory — the iconic AGENTS.md that carries your project instructions plus a MEMORY.md that encodes what the agent learns (e.g. “run the tests with pytest -x, never the full suite”). Plain files, deliberately no memory database and no codebase index: the repo is explored dynamically with grep. Just-in-time reads beat a stale heavy index.

Skills — the omnipresent skills feature that encodes reusable workflows loaded only when invoked, so the context window doesn’t get bloated with all your instructions at once. The repo already ships a set of skills under .decode/skills/ that you can try out of the box.

LSP Server — a key component of a coding agent. It keeps an index of all your variables and functions. The loop gets syntax and semantic information about the codebase through 2 channels instead of brute-forcing through it or waiting for the code to run. On demand, the model asks where a symbol is defined, who calls it, what its real contract is, and what’s broken in a file. The model gets 1 precise file:line answer. After each edit, a fast syntax checker feeds its findings back into the loop. The agent fixes a type error in the same turn, before running/compiling any code. The cheapest feedback loop in the system.

Compaction — not a module but a behavior of the harness itself that helps keep the context window as small as possible. The most iconic strategy is to compact the window once it goes over a threshold, squashing the window’s old head into a summary + keeping the recent interactions fresh: [summary, *tail]. You can also call this manually via /compact or simply clean the whole context window via /clear.

That’s the headless harness. Now, let’s plug it into the interfaces you use every day.

The Interactive Mode

The first is the TUI: the interactive mode, a terminal interface wired directly to one live session of the headless harness, in memory, in the same process.

We built it in Python, where prompt_toolkit handles input and Rich handles output of the terminal experience. It’s append-style, like a conversation log, rather than a full-screen app. Keystrokes go in, and the session’s event stream comes back in real-time via Python generators: streamed text, tool calls, approval prompts.

Why Python? To be honest, we picked it mostly for teaching purposes.

From my recent poll, I realized that all my audience knows Python. So this makes it accessible to you. But if I were to ship this for real, I’d reach for TypeScript (the language claude-code, OpenCode, and Pi chose) or Go, which compiles to one tidy binary. Still, Aider proves Python can carry a serious coding agent.

And in this series we will focus on the “why”, the design and architecture decisions. Thus, the programming language is not that important and you can easily swap it out for another language.

The steering queue

What happens when you send a new command and the agent is already mid-task?

Injecting it the moment it’s sent would corrupt a tool call already in flight. Ignoring it makes the agent unsteerable. Every serious harness hits this: Pi drains steering messages mid-turn but follow-ups only at the turn boundary, and claude-code ranks queued input so user messages never starve.

Decode’s answer is the steering queue + priority gate. Input is buffered the instant it arrives into the steering queue and injected only at a safe boundary: before the next model call, never mid-tool-call. The gate decides how to process the messages from the queue: steer now, answer later, or abort.

The steering queue can be a simple in memory FIFO queue. No need for a complex queue infrastructure like RabbitMQ.

You can already clone the Decode repository and try it out for yourself.

You have all the setup instructions in the repo. There are 0 costs to try it out. We recommend starting it with Modal, which offers30 in free credit every month.

The experience is similar to what you already know: you run decode in any repo and the TUI opens as a fresh session.

We bundled together a few demo’s as skills under .decode/skills/, so you can easily try out the coding agent with 0 friction:

Such as creating a snake game:

Fetching data from a GitHub repo and rendering a web page:

Or something more complex like scraping data from the internet, extracting a knowledge graph of entities and topics and rendering them into a cool visualization:

What if we want to run the agent remotely?

The Remote Mode

Remote mode keeps the harness headless and runs it on a server through an agent runtime. We use Kitaru, the agent runtime built by ZenML, which provides durability, replay and distributed HITL features. Then we ship the Kitaru control plane to GCP via its open-source option. From there, it orchestrates and scales the remote agents, which run on Modal.

This setup lets you run background jobs, scheduled tasks, and full automation pipelines: grab a ticket from Linear, launch 5 to 10 implementations in parallel (each with its own PR), and let a frontier foundation model judge the candidates and rank them for you.

A runtime gives the headless harness a control plane: deploy it once, then trigger, monitor, and feed it input remotely, without building any of that plumbing yourself.

The infrastructure’s moving pieces can get confusing. There are 3 main pieces:

• The Control Plane: The runtime that coordinates the headless agents. Powered by Kitaru. Super similar to orchestrators such as ZenML, Prefect or Temporal. That’s why ZenML created Kitaru, specialized in running agentic workflows. Using Kitaru’s open-source Docker image, we will deploy it to GCP.

• The Execution: Where the headless agents run: Python environment (local) and Modal (remote).

• The Sandbox: Where the tools execute: Docker (local) and Modal (remote).

The runtime records the run’s progress step by step, so when a sandbox dies mid-task, the run resumes from its last recorded step instead of restarting.

Same reasoning for human input: the run freezes at a question, consuming no compute while it waits, and resumes the moment you answer, hours later or from another terminal.

Because every step is recorded, a finished run can be replayed with one thing changed (a different model, a fixed prompt) against the original as the baseline.

Try it out

You can run the remote version by going to the course repo, typing decode run <your_command> in your terminal, and opening Kitaru to watch the agent work.

Two modes, one core. That raises the obvious question: why not both at once?

All the Modes, One Message Bus

We keep Decode simple by keeping the headless harness as the core business logic that can be served in two modes: local and remote.

But you can take this further. OpenCode uses a message bus between the headless harness and every interface to synchronize sessions across their TUI, IDE, WhatsApp, web app, mobile clients. So they all drive the same live session: when one interface publishes a message, all the other ones are instantly synced.

Whichever mode runs it, one question decides whether any of this is real engineering: can you prove the agent works?

The Observability and AI Evals Layer

The evals layer answers 3 questions, each with its own mechanism.

Internal benchmarks answer does it work? These are the metrics you try to beat with every new feature. You can use public benchmarks such as Terminal-Bench to compare your agent’s performance against others. But the real deal is to build a custom benchmark suite from your own tasks that optimizes for your specific use case.

Each case is a task instruction, a fresh sandboxed environment, and a hidden oracle test that decides pass or fail. One example case is “add a --json flag to decode run“, with a hidden pytest asserting the output parses. It’s the same pattern SWE-bench applies to real GitHub issues and Terminal-Bench applies to terminal tasks.

Your own tasks predict your agent’s usefulness better than any leaderboard.

Regression tests answer does it still work? Every new feature re-runs a regression suite and compares scores against the baseline, so a prompt tweak or a new tool can’t silently make the agent worse.

Production evals answer does it keep working? Every session is traced with Opik, Comet’s open-source LLM observability platform: each model call, tool call, and token count is a trace, while the conversation is logged as a thread. Live scoring tracks quality in production, where you run a bunch of AI Evals metrics on sampled traces to detect unwanted behavior to create warnings or alarms.

We will also use Opik as our evaluation harness to track our eval datasets and run a coding agent on all the scenarios from the benchmark and regression tests.

Next Steps

You’ve seen a bird’s-eye view of what a coding agent looks like: one headless core, a small tool-calling agent inside a much larger harness, 2 modes, and an evals layer that measures performance.

In the next lesson, we get into the code and start building the agent loop hooked into the TUI.

Here is the course roadmap, lesson by lesson (see all in GitHub):

1. Building a Coding Agent From Scratch ← you are here

2. The Bare-Bones Coding Agent Loop

3. From a Raw Shell to a Sandboxed Coding Agent

4. Context Engineering for Coding Agents

5. Agents Catalog, Subagents & Parallel Fan-out

6. Remote Headless Mode & Durability

7. AI Evals Foundations: Benchmarks, Regression and Online

8. AI Evals on Steroids via Replays

Or the video lecture supporting the first two lessons:

But here is what I’m wondering:

Which part of your daily coding agent is still a black box to you: the loop, the sandbox, the memory, or something else?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

Special thanks to Modal, Opik (by Comet), and Kitaru (by ZenML) for sponsoring this open-source course and keeping it free!

Images

If not otherwise stated, all images are created by the author.

Agent memory is one of the hottest problems in AI engineering right now, with Graphiti, mem0, HydraDB, and cognee all racing to solve it. A race this crowded means one thing: nobody has cracked it yet.

But if you’re thinking of jumping straight into an agent memory tool without understanding how it works under the hood, read this first.

Here is what happened in one of my tests: LangChain’s MongoDBGraphStore gave me a knowledge graph (KG) in 10 minutes, and from 5 documents it invented 17 node types and 34 relationship types, where part_of, Part Of, and part of were three different ways to express the same relationship.

That’s why it’s incredibly important to understand how the memory layer works under the hood, so you know its limitations, how to properly configure it for your data, and how to plug it into your agent.

Or even decide if you need a unified memory at all, and if so, which tool to use.

Claude Code comes with a built-in memory layer powered by files. Is this enough for you or not?

Then, if you move to a more powerful solution, should you use a vector database, a graph database, or a combination of both? Do you need temporality or versioning?

And ultimately, how should you serve your memory layer to your agent? Through an MCP server, a CLI, or plain skills?

The reality is that there is no blueprint on how to build your own unified memory layer. Too many choices, too many trade-offs.

But that’s why I want to show you how to build a unified agent memory from scratch, in the most complicated way possible: via knowledge graphs (KGs). Plus all the other moving pieces: ingestion, querying, and serving.

Why? Because all the popular vendors (cognee, Graphiti, Neo4j’s agent memory, etc.) go in this direction. Plus, if you know how this works, it will be incredibly easy for you to build simpler solutions.

So, I am not saying that KGs are always the way to go. They are not. But they are definitely something you need to understand. At least to know when to stay away from them.

Let’s go!

A Session With a Unified Memory

First, here is what I expect a unified memory plugged into a harness to do.

I open Claude Code and ask what I need to do today. The memory already knows: building a coding agent for my next open source project.

I ask it to gather everything relevant I have on building a coding agent: Obsidian notes, Readwise highlights, starred repos, all ingested long ago. Then to build an LLM wiki on top of that as my agent memory while I implement the plan.

When I’m done, the conversation is ingested too, so the next session already knows all my decisions around the project.

Nothing in that session lived in the harness. Claude Code was just the interface.

So how do I implement a unified memory that supports this? Let’s kick off with the architecture.

The Architecture, End to End

Sources in, subgraphs out. In between sit 4 pieces: an ontology, a write path, one collection, and a serving layer.

The whole database layer is powered by MongoDB: a document warehouse plus a knowledge-graph store doing text, vector, and graph search in one collection. Because we use a single database, lineage is free: nodes hold references to their source documents, not copies from a separate source. This works incredibly well at 2-3 hops at query time.

OG databases such as MongoDB scale really well up to millions of documents via sharding and replicas.

Writes are 2 durable pipelines. A data pipeline normalizes each source into the warehouse. A memory pipeline cleans, chunks, extracts, normalizes, embeds, and writes the knowledge-graph objects. Both pipelines are powered by an orchestrator, such as Prefect or DBOS, to serve them, scale them, and make them durable.

On the read side, we offer multiple ways to interact with the unified memory: standard graph search encoded directly into the tool, agentic search where the agent writes its own MongoDB query, and a deep-search primitive that creates an on-demand LLM wiki for exploring larger subgraphs via progressive disclosure. Reads never touch the orchestrator, as they need to be fast and don’t benefit from the orchestrator’s durability or scheduling features.

The memory is reachable only through a FastMCP server exposing agent-shaped primitives (query, deep-search, ingest), never raw database operations. The MCP server should never be an API wrapper, but directly expose business logic to the harness. A hook fires the conversation ingestion as the session runs, so what the agent learns flows back into the memory — a process known as “continual learning.”

The most interesting part is not the database, the ingestion, or the query logic, but the ontology that makes everything possible.

The Ontology Is the Contract

An ontology is a contract between the writer and the reader. The LLM extracts against it. The query layer reads against it. Define it once as Pydantic, serialize with model_json_schema(), and dump that JSON into both writing and reading system prompts. 1 artifact, 2 consumers, no drift.

The ontology defines what the knowledge graphs look like. All the KG instances are extracted by passing the input data and the ontology schema to an LLM.

The memory is 3 layers. The long-term layer is the knowledge graph: the POLE+O entities plus preference and fact. A short-term layer holds the dialogue as conversation and session nodes. A reasoning layer records the agent’s work as agent, tool_call, and memory nodes.

One powerful ontology design is based on the 5 POLE+O nouns from law-enforcement analysis: Person, Object, Location, Event, Organization, which Neo4j Labs writes up as the POLE+O data model. Each noun carries an optional subtype used to further refine the ontology within a specific domain. For gaming, this can look like:

• Person: Player, Character

• Object: Item, Quest

• Event: Raid, Battle

Because the agent memory is often used for a personal assistant, you also need to plug preference and fact nodes into the ontology. A fact is an atomic subject-predicate-object triplet, an island no edge may touch, reachable only by similarity. A preference is the personalization layer with typed slots.

nodes: person · organization · location · event · object
       preference · fact
edges: related_to, with semantic_type ∈ {
         knows, member_of, employed_by,
         owns, uses, located_at, resides_at,
         alias_of, has_task, ... 
       }

Now, on top of the KG instances extracted by the LLM, we can also have structured nodes that are directly inferred via code.

Such as:

nodes: document · chunk
edges: part_of · next · mentions · referenced · has · same_as · superseded_by

There is a lot more to talk about ontologies. I have a full article just on this.

With the contract fixed, let’s see how we can ingest KG objects into the unified memory based on the ontology schema.

The Write Path

There are 7 stages in one direction. Chunk the document (512-token chunks, 64 overlap). Extract nodes and edges with an LLM. Validate. Resolve names. Embed. Deduplicate. Upsert into the MongoDB unified memory.

During the LLM extraction, we pass in a chunk and we receive a JSON response with nodes and edges, the same shape Neo4j Labs lands on in How Entity Extraction Works.

The model sees one chunk and nothing else. It receives no IDs and no prior state. It returns JSON. Every edge endpoint points to a node’s name in the same record. This allows us to scale the extraction pipeline through batching to process large documents of 1,000,000+ units.

Validation is done via Pydantic. We load the JSON response into a Pydantic model and validate it against the schema.

Name resolution only checks whether two canonical names match. It’s done purely on the name and its aliases: alias, exact, fuzzy 0.85, semantic 0.80, filtered by type.

But there is a next step, where we actually check if an entity is the same instance as another that already exists in the graph.

Deduplication runs on embeddings computed on the whole node content: ≥0.95 cosine merges, ≤0.85 makes a new node, and the gap writes a same_as edge for a human to judge.

So why two steps?

Name resolution simply helps us group entities by their canonical name. For example, “Demis Hassabis” and “Demis Hassabis, CEO” would both be resolved to “Demis Hassabis”. Or “Apple” and “Apple Inc.” would both be resolved to “Apple”. Super powerful for analytics and human curation.

Deduplication is the sensitive one — it decides what actually merges. It shouldn’t flag “Apple” and “Apple Inc.” as duplicates, as they are different entities with different meanings.

A wrong merge is the only unrecoverable mistake. That’s why, when the similarity score has low confidence, between 0.85 and 0.95, we don’t merge the nodes automatically, as the operation can be irreversible.

I break the full algorithm down in Keep Your Knowledge Graph Clean.

Which leaves the question that decides your RAM bill: once the triplets are extracted, how do we store them in the unified memory?

Append-Only Logs vs. a Single Knowledge Graph Collection

There are 3 designs, and the winner? Well... It depends... As always.

An append-only log plus a materialized view buys versioning and soft-delete for free, but you pay for it. Nested relationships duplicate edges and make them awkward to query. Separate edge docs give queryable edges, no duplication, and the cheapest footprint, at the cost of built-in temporal history and versioning.

Because the graph is a projection of the append-only log, a bad extraction is fixed by invalidating its event. You can change the materialization logic and replay it without re-extracting a single source. Also, full-graph temporality belongs to the log alone.

The log’s real cost: a vector index is at least as large as the data it covers. Indexing both the log and the view inflates ~10 GB of data toward ~40 GB of RAM. Index only the view, leave the log cold on disk, and it collapses back toward ~10 GB. But the idea is that adding an append-only log brings a lot of engineering effort you then have to maintain.

So think three times before you sign up for it.

At query time, the agents see the materialized view, not the log itself. Given the RAM cost, reach for the log only when versioning or the time dimension is core to your business logic. Otherwise, keep it simple and stick to a single knowledge graph collection.

Within the data model, Node IDs are {user_id}:{type}:{name}. Edge IDs are {source}|{type}|{target}. Because the key is content-derived, every write is idempotent.

The reality? You can get temporality even without a dedicated append-only log. Preferences and facts carry valid_from / valid_until, and a superseding preference writes a superseded_by edge.

Three Ways to Read the Memory

The 3 retrieval methods below are the serving layer’s search primitives, the tools the MCP server exposes to the agent.

Graph search is the default retrieval, baked into a single function. It runs $vectorSearch and $text in parallel over the KG collection. Fuse the ranks with Reciprocal Rank Fusion (RRF). Keep the fused top 10 as seed nodes, then expand with a bidirectional $graphLookup at 2 hops by default. That multi-hop step is the entire difference between RAG and GraphRAG.

Deep search widens the walk to 3 hops and persists the result to disk as a memory wiki the agent can access via progressive disclosure. The LLM wiki is a mini-graph of its own: you keep the (entity, relationship, entity) triplets by cross-referencing files, where each file is an entity. You can see it as a local cache that is either discarded between sessions or kept in sync with the KG collection.

NL query is agentic search that lets the LLM write its own MongoDB queries based on the ontology. Here you have to validate that the query is syntactically correct, and especially guard it against harmful intents such as deleting or modifying data. It’s safer to use this mode purely for read operations.

RRF replaces a reranker (a second model that re-scores the retriever’s top candidates) you don’t need yet. It is 20 lines of Python, needs no model, and runs in microseconds. Reach for a cross-encoder (a heavier model that compares the query against each candidate) when you search millions of heterogeneous documents.

When do you need to switch from a single database (MongoDB) to a graph database (Neo4j)? Past 4 hops, past ~100M to 1B vectors, when graphs are your business logic, or purely to visualize them: that is where you leave MongoDB for Neo4j. It’s common to use a single database in production (MongoDB) and a graph database as an internal tool for data exploration and analysis (Neo4j). This is a powerful combo, because for an internal tool you don’t care that much about 99.99% SLAs, while you can fully leverage Neo4j’s Cypher query language to explore and analyze your graph data.

Open, Closed, and Small

Extraction (from ingestion) and query translation (from agentic search) use gemini-3.1-flash-lite. Embeddings are Voyage voyage-3.5 at 1024 dimensions. Both are closed APIs.

You can take this further and fine-tune an open-source SLM (such as the Liquid family of models) as an extractor, query translator, or embedding model on your own data.

The issue with SLMs is that without fine-tuning they perform poorly on new data. But the beautiful part about them is that after fine-tuning they perform as well as a frontier model on a scoped task, while being much cheaper and faster to run.

I won’t go too much into fine-tuning here, but the idea is that at first you always want to start with an API that quickly gets stuff done, and then slowly replace the most important parts with your own fine-tuned models. You usually have to decide this based on cost, latency, performance, data privacy, or other dimensions important to your use case.

Build vs Buy: When Not To Build This

Everything above assumed you build everything yourself. But in reality, there are 3 levels to choose from.

Level 1: build the memory, the logic, and the serving layer yourself on one database.

Level 2: keep only the business logic and serving layer on top of a memory SDK, such as Graphiti, neo4j-labs/agent-memory, or mem0.

Level 3: run an off-the-shelf engine, such as cognee, managed Zep, or HydraDB.

This is a problem about owning your context layer, which I have a full article on.

Buy when you don’t care that much about customization. Build when you want your own solution. But since building everything yourself usually isn’t worth it (or you just don’t have the time to do it — even with Claude Code), the best solution is Level 2, where you own the business logic and can customize the serving layer to your needs.

But honestly, because things move so fast and I am lazy, my long-term memory still lives in Obsidian, Readwise, and Google Drive, plus a serving layer that creates scoped LLM wikis per project as agent memory. This way, you create mini-graphs for one specific problem, powered purely by .md files, without any infrastructure hassle. The reality is that this works for personal use only, and not so much for shipping a production-ready solution. I write about that approach in LLM Wikis as Agent Memory.

Still, I am applying the strategy based on KGs and ontologies to build Pulse, a data mining tool that runs the exact same strategy presented above on a knowledge-work ontology to extract the top 0.1% signal from the AI industry. I want it custom because I want it to be super light, powered by LadybugDB, and to leverage my Claude subscription for the extraction and deduplication part, built as a hybrid between skills and Python modules.

But here is what I’m wondering:

If you keep dumping everything into the unified memory, where do you draw the line between what your agent should remember forever and what it should be allowed to forget?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

Special thanks to MongoDB for sponsoring this article and keeping it free!

Explore Next

1. Iusztin, P. (n.d.). Own Your Context Layer: Portable AI Agent Memory. Decoding AI. https://www.decodingai.com/p/the-context-layer

2. Iusztin, P. (n.d.). Ship a Knowledge Graph Ontology in 5 Minutes. Decoding AI. https://www.decodingai.com/p/ship-a-knowledge-graph-ontology-in-5-minutes

3. Iusztin, P. (n.d.). How to Keep Your AI Agent’s Knowledge Graph Clean. Decoding AI. https://www.decodingai.com/p/keep-knowledge-graph-clean

4. Iusztin, P. (n.d.). LLM Wikis as Living Memory for AI Agents. Decoding AI. https://www.decodingai.com/p/llm-wiki-agent-memory

5. Neo4j Labs. (n.d.). Why Neo4j? Graph-Native Memory Architecture. Neo4j Agent Memory. https://neo4j.com/labs/agent-memory/explanation/graph-architecture/

6. Neo4j Labs. (n.d.). POLE+O Data Model. Neo4j Agent Memory. https://neo4j.com/labs/agent-memory/explanation/poleo-model/

7. Neo4j Labs. (n.d.). How Entity Extraction Works. Neo4j Agent Memory. https://neo4j.com/labs/agent-memory/explanation/extraction-pipeline/

Images

If not otherwise stated, all images are created by the author.

I spent 18 months turning my second brain into my living research memory. My digital life contains 10,994 notes: over 5,000 in Obsidian, another 5,000 in Readwise, plus more in Notion and Google Drive, growing by roughly 250 a month.

The reality is that most of my notes and bookmarks transformed into a graveyard. When I start a new article or open a codebase, I can’t recall the high-signal notes I already have, so I work without them or burn an hour digging.

The reflex is to reach for Codex or NotebookLM. But those are harnesses: they sit on top of your work, and the moment the chat ends, the context window loses all of it.

More context isn’t the fix. You re-paste the same links and re-explain the same project every session, while past research sits inert in a vault your agents can’t reach.

So , CTO of Towards AI, and I built the alternative, and we’ll show you how: your own AI Research OS. It’s a memory layer that sits between your second brain and any harness (Codex, Claude Code, or your own). It runs deep research across your notes and the open web, and stores what it finds as an LLM wiki you can query, maintain, and grow.

We first touched on this subject in a talk at the AI Engineer World’s Fair. After we saw that people loved it (it was published as a keynote in the online tracks - out of 82 videos - and is the 2nd most popular one), I decided to write an article about it as well.

By the end you’ll learn how to turn your 10,994 notes into a queryable LLM wiki that your AI agents can use as their agent memory, useful for research, coding and content creation.

You will build an LLM wiki as in the image below, where the payoff already compounds with just 12 sources:

Or go crazier with a larger wiki:

By the end, you will also see why the wikis don’t contain all my 10,994 notes to keep ingestion costs low. Let’s go.

Why a Bigger Context Window Won’t Save You

Most research doesn’t need this system. If you want a quick answer or a one-off, Google it, or reach for Codex and Claude Code. You need this only when the research has to stick and an agent has to reuse those sources later.

You don’t need a giant vault either. This is about topic density, not size. It already pays off on a topic-sized cluster: around 20 notes on one subject, or 2 or 3 repos, like coding agents such as Pi, OpenCode, and Aider. Louis-François runs it on a few hundred notes, not my 10,000+, and it still earns its keep.

Why not NotebookLM? It’s great at digesting sources, but you don’t own it, you can’t personalize it deeply, it isn’t agent-native, and it’s weak for coding since it’s browser-bound.

Why not a vector-database RAG pipeline? That’s the right call at production scale, but it’s infrastructure. It’s hard to inspect or edit by hand, and overkill for a personal tool you open every day.

So you build it yourself: a personalized research assistant that uses an LLM wiki knowledge base that’s light and human-friendly as its agent memory. The perfect context engineering technique as an interface between the human and the AI agent.

Now, I want to go over, step by step, how we reached our final deep research + LLM wiki design, with the “why” and “how” in mind.

The Deep-Research Loop, Version 1: Mining the Public Web

A year ago, building the Agent Engineering course, we scoped the loop tightly: give it a topic plus a few hand-picked golden links, and get back a single static research.md.

The deep research algorithm first scrapes the golden links for seed context. Knowing them up front lets the agent frame better questions. Then it runs query rounds: one orchestrator generates the questions, and a sub-agent per question searches (Gemini grounded in Google) and returns links plus a summary, which the orchestrator aggregates so the context never explodes.

Three rounds of 6 queries surface 40–50 links, which is too noisy to keep whole. A ranking step scores each source against the topic. Only the top-K get fully scraped, the rest are kept as summaries, and everything is compiled into one flat research.md. That version generated 35 course lessons fast.

It worked for the course, but it was aimed at the generic public web, with every golden link hand-picked from our own second brain.

Version 2: Point the Loop at Your Second Brain

Same loop, new target: aim it at your own sources instead of the public web, where you’ve already filtered what matters. Your second brain is a curated set of golden links, so the input shrinks to just a topic and the loop finds the rest.

You plug in Obsidian, Readwise, NotebookLM, and GitHub Stars, then extend with Gemini Deep Research, YouTube, Google Drive, or Notion. Code is a special case: a GitHub sub-agent clones a repo and builds a high-level architecture note.

The token-efficiency trick lives in the reranker. It scores each candidate from 0 to 1 against your question using only its metadata and summary, never the full text, then passes only the top-K to the model.

The new problem is that the output is frozen. You still end up with a static research.md, and real research isn’t static. You want another question answered, or part of it goes stale, and re-running the loop from scratch is expensive in tokens and time. The fix is a new layer that sits on top of the raw data: the LLM wiki.

Version 3: From a Static Pile to a Living Wiki

The pivot came from Andrej Karpathy’s idea of LLM-maintained knowledge-base wikis [1]. Instead of re-deriving knowledge from raw documents on every query, which is the RAG pattern, the LLM incrementally builds and maintains a persistent, interlinked wiki between you and your sources.

Since I realized the power of LLM wikis, I started to use them for all my personal setups as my agent memory. It’s simple, powerful and beautiful.

Since giving this talk, we found that Google had published its Open Knowledge Format (OKF), an open spec that “formalizes the LLM-wiki pattern into a portable, interoperable format” [2]. It’s built on the very blocks you’ll see below: plain markdown with YAML frontmatter, an index, a log, and links that form a graph. When Google independently ships the same architecture as a standard, you know the direction is right.

During ingestion it generates, as a byproduct, derivatives over your raw sources: entities and concepts, comparisons between them, notes tied to your questions, open questions, and a synthesis:

Storing each finding as an individual raw file, instead of one flat research.md, is the one change that makes everything downstream queryable and growable. Sources can include Obsidian, GitHub, Google Drive, even custom URLs (plain curl for simple sites, Bright Data for bot-walled ones).

The natural fear is that “index + wiki + query” means vector databases and knowledge graphs. It doesn’t, and that’s the surprising part.

A Memory Layer Built From Plain Files and No Database

Vector databases, knowledge graphs, and semantic and text search all add real complexity, too much for a personal research OS. So drop that infrastructure and build the whole thing on files and references: no database, just a simple index rooted in how your filesystem already works.

The index is the retrieval layer. An agent reads a single index.yaml first, a catalog with a summary and metadata for each source (original file, origin, title, authors, date).

There are 3 layers. A raw folder holds the immutable source data, a wiki folder holds the LLM-generated derivatives, and the index points to all of it. A real example is an index.yaml cataloging 10 sources and 38 wiki pages.

You even get a knowledge graph for free. Because everything is Obsidian-flavored markdown with references, Obsidian’s local graph renders the connections out of the box: entities like OpenCode or MCP linked to concepts like tool registry, context compaction, and sandboxing, and the sources they touch.

Obsidian is only a viewer, though. The whole system runs from any working directory through Codex or Claude Code, so the graph is a bonus on top.

Querying the Wiki, and Why It Never Freezes

The agent queries through progressive disclosure. It starts at the index.yaml summaries. If it needs more, it opens the source wiki page (an expanded summary), and often stops there.

If not, it follows references into the derivatives (entities, concepts, comparisons, notes). Only as a last resort does it read the raw source. Summaries are computed once, at ingestion, so the context window stays small.

The wiki is also alive. Ask a question and the LLM can spawn a new concept, note, or comparison, and each question is written to a log. So the wiki evolves as you talk to it, not only when you ingest data.

It’s never frozen, and it grows 3 ways: ingest a new custom link, run another deep-research round, or let it create derivatives purely from your questions.

One design choice makes this safe to run against a decade of notes: the wiki never sits directly on your whole second brain.

Scope It to a Project With PARA

Your second brain stays an immutable snapshot. I organize everything with the PARA method: Projects, Areas, Resources, Archive [3]. Sources are piped into Resources as a flat list, then referenced into Projects and Areas. So Obsidian stays read-only; I don’t want the LLM editing notes I write by hand.

The wiki is scoped per project, not global. When you start something new, you reference that snapshot through the deep-research loop and scope it down to the project, running the loop or ingesting specific repos, articles, and notes via skills plugged into a harness.

Scoping is a scale choice as much as a safety one: plain files stay fast and inspectable at project size, but a whole-vault wiki of thousands of documents is exactly where a vector or graph database finally earns its place.

That’s the ~100-note wiki from the top, kept small by scoping to one project, yet reaching my entire 10,000+ note second brain through the loop.

The mental model is that the project is the work and the second brain is the research. A project is anything where you turn research into work: an article, a video, a set of slides (this talk was built this way), or a whole codebase. That’s the whole architecture, and the fastest way to see why it matters is to watch it run.

See It Run: Three Demos

Everything ships as one open-source Claude Code plugin you can clone and run on your own notes (https://github.com/iusztinpaul/ai-research-os-workshop), tweakable for any harness.

It’s four skills: /research (build or query the wiki), /research-distill (a per-piece research.md), /research-lint (health-checks), and /research-render (slides or a brief).

Demo 1 is research from a brain dump around “agentic harnesses”. You point the research skill at a file of notes plus a few must-include references. It scrapes those for seed context and picks a depth: fast (1 round of 3 queries), light (2 rounds, 3 then 2), or deep (3 rounds of 3). Out come the raw files + the LLM wiki:

Demo 2 is ingesting and comparing GitHub repos. You give it 3 harness repos (OpenCode, Pi, Hermes), no deep research, scoped to architecture, sub-agents, memory, and the permission flow. It clones each, writes per-repo notes, then builds cross-repo comparisons.

Demo 3 is ad-hoc links on GraphRAG, then questions to create notes. You hand it 3 arbitrary links to build a wiki, then ask how agentic GraphRAG differs from a plain knowledge graph. It answers off the wiki and updates it in place.

It was super hard to show the full demos here. You can find the full examples within the Examples section of the repo or watch me go over them in Obsidian and Zed in the full video.

Where This Is Going

Some connectors are missing on purpose (Google Drive, Notion, Slack) — the point is for you to add what you need. Louis-François added YouTube transcript ingestion in seconds with one prompt to Codex.

Two weaknesses stand out. The byproducts are often poorly written, fixable with the article writing profiles I already use. And as data grows, the LLM makes subtle errors: concept confusion (comparing Claude Code and OpenCode, it merged how each handles tool calls into one explanation) and superficiality (the Claude Code queuing system took many follow-ups to get right).

The fix is constant linting, which is what /research-lint does: it sweeps the wiki for orphan sources, broken links, stale claims, and contradictions.

What’s Next

Everything you saw here is open source. Clone the repo (or install it as a Claude Code plugin) and run it on your own notes today: https://github.com/iusztinpaul/ai-research-os-workshop

If you’d rather see the whole system in action first, watch the full talk and I gave at the AI Engineer World’s Fair:

And if you want to go beyond a workshop repo, such as building a deep-research-plus-writing multi-agent system from scratch and shipping it to production with AI evals and an observability layer on top, that’s exactly what we teach in our Agent Engineering: Building Multi-Agent Systems course.

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

Explore Next

1. Karpathy, A. (n.d.). LLM Wiki: A pattern for building personal knowledge bases using LLMs. GitHub. https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f

1. Google Cloud. (n.d.). How the Open Knowledge Format can improve data sharing. Google Cloud Blog. https://cloud.google.com/blog/products/data-analytics/how-the-open-knowledge-format-can-improve-data-sharing

2. Forte, T. (n.d.). The PARA Method: The Simple System for Organizing Your Digital Life. Forte Labs. https://fortelabs.com/blog/para/

Images

If not otherwise stated, all images are created by the author.

Models are commoditizing fast. Harnesses already have. What you care about is your research, your notes, your conversations, your tasks, your preferences, and your domain knowledge.

“Free” open-source harnesses don’t make you free. What you want to own is the context layer. Every agent is the same model + runtime + harness underneath, rebuildable on an open stack [1].

I hit this myself. The deeper I built into a single harness, the clearer it got how much I’d lose the day I had to leave. The loss comes in 3 forms.

Failure 1 — You start from scratch. You’ve run Claude Code for 6 months. You switch to an open model and every past conversation, every preference the agent learned about you, is gone. You begin again from zero.

Failure 2 — Your skills are hostage. The harder lock-in is your business logic. If your skills and workflows are coupled to one harness’s features and keywords, switching doesn’t just lose chat history — your custom logic breaks or quietly performs worse on the next tool.

Failure 3 — You’re billed at their mercy. The plan you depend on can be pulled, suddenly gated behind expensive pay-as-you-go, restricted from the latest models (e.g., the Fable story), or repriced from 200 to 1,000 overnight. With high switching friction, you can’t leave. You never know how the AI world can change. But for sure it won’t stay as is now.

So what do you do?

Keep your context layer detached from the harness. Any harness, any model (open or closed), plugs in and within ~5 minutes knows who you are, what you’re working on, what matters to you and how you like things to get done.

Everything that follows is based on the two projects I am working on: Scrabble, my own context layer anchored into my Obsidian notes, Readwise library, Notion, and Google ecosystem; and Tree, the project I am working on with for our next book, where we teach how to build a personal AI assistant from scratch.

The Architecture of Your Context Layer

A context layer is made out of 3 core components. First, a unified memory that holds everything you know. Second, a serving layer (MCP server or skills) that is the interface to that memory and the business logic that defines how the memory is used. The harness sits on top and is deliberately disposable.

The unified memory fuses what used to be separate systems: a filesystem, keyword search (BM25), semantic vector search, and a knowledge graph of typed POLE+O nouns (Person, Organization, Location, Event, Object), all in one place. The goal is a memory that belongs to you.

What I always preach is to try your best to build your unified memory on top of a single database that supports text, semantic, and graph search (such as MongoDB). Start simple and add complexity only when your use case demands it.

The serving layer has 2 angles. The first is an MCP server, surfacing Tools, Resources, Prompts, Skills, and even MCP Apps (e.g. a graph visualizer) [2]. It wraps the business logic for how memory is queried and updated. That’s what makes it portable.

The second form skips the server entirely, which is based on skills shipped straight on the filesystem (an AGENTS.md plus a folder of skills). Skills are an interface too. They’re leaner, with nothing to run, but more tied to a given harness’s conventions. It’s how Scrabble’s wiki layer gets consumed.

In practice the strongest setups do both: a portable MCP server for memory access, skills layered on top for higher-level workflows [2]. To avoid fragmenting your business logic, host the skills directly on your MCP servers, coupling them with the rest of the server’s business logic.

In Tree, where I am building a unified memory powered by knowledge graphs, everything is discovered as ordinary MCP tools. That is exactly why “swap the harness, keep the memory” is a one-line config change, not a migration.

Here is what happens when you interact with the context layer:

You enter a prompt into the harness, which becomes an MCP tool call to the server. The server queries the unified memory, and results flow back into the agent’s context. Nothing crazy so far.

The interesting part happens when you start adding stuff to your memory. It’s where continual learning happens.

Building a Unified Memory for Continual Learning

At a high level, the pipeline is simple. Ingest data into the unified memory → extract entities and relationships into a graph → index for hybrid and graph retrieval → expose everything through an MCP server → connect it to a harness like Claude Code or OpenCode. Building the knowledge graph was the easy part. Designing how the agent interacts with it was the hard part.

Don’t expose raw database operations. Agents struggle with them. Instead, design the server around how agents actually search and write, giving them high-leverage primitives they can compose like LEGOs inside skills. In Tree that’s 6 tools: 3 to search, 3 to write.

There are 3 search tools. nl_query_memory is the default — an LLM maps a natural-language (NL) question into a MongoDB hybrid-search + graph query. query_memory is the deterministic fallback for structured filters when NL fails. deep_search_memory handles large result sets (50+ docs) via progressive disclosure: it writes intermediate results to a YAML index creating a localized LLM wiki on the fly.

The distinction between nl_query_memory / query_memory and deep_search_memory is important because when retrieving chunks from the unified memory, you need to further compress them before adding them into the context window to avoid exploding your costs. But that means you lose a lot of information. Thus, creating a wiki on the fly trades latency for performance and lower costs.

There are also 3 write tools: ingest_url, ingest_file, and ingest_conversation.

The last is auto-triggered by a hook that ingests incrementally every ~10 conversation turns and finalizes when the conversation ends. That hook is what makes the learning loop continual: the system writes back what it just learned without you asking, so it gets smarter the more you use it.

And “smarter” is concrete. Across tens of thousands of notes, it finds links between things you’d long forgotten you had, connections you’d never have made by hand. It’s how you finally harvest your own past work and your graveyard of bookmarks.

Now, let’s see how to actually build the unified memory layer.

Build It Three Ways

You’re implementing 3 pieces: the unified memory, your custom business logic, and the serving layer. How much you build yourself is also split into 3 levels of effort, which translates to a build vs. buy discussion.

Level 1: Build all three from scratch on a database that supports everything at once (such as MongoDB), implementing the memory and the MCP serving layer yourself. Maximum control, maximum effort.

Level 2: Build only the business logic and serving layer, on top of an SDK that already implements the memory layer: Graphiti, a bi-temporal entity/fact/episode graph; Neo4j-Labs’ agent-memory, a graph-native POLE+O store; or mem0, a vector-first layer that stores facts as rows.

Level 3: Off-the-shelf memory engines you mostly just run: cognee, an Extract-Cognify-Load pipeline of typed nodes that are graph + vector at once; or managed services like Zep (temporal context graphs on the open-source Graphiti engine) and HydraDB (a graph DB on tiered object storage, pitched as “own your memory, no data leaving your stack”). Lowest effort, least control over the data model, and a vendor dependency. Cognee can import and export memory across Mem0, Zep, and Graphiti if you need to switch.

Now zoom into Level 1, the from-scratch path. Building the memory yourself splits into 2 versions. Version 1 (more complex), such as Tree, the agent memory I am building, which is a GraphRAG system whose retrieval fuses vector and keyword search with knowledge-graph traversal. It contains a POLE+O ontology, the index (graph + vector + text), the data and memory pipelines that ingest into it, and the query algorithms that read it.

POLE+O is a popular data model for building ontologies, borrowed from law-enforcement and intelligence analysis. Its beauty is that it’s perfectly balanced. It’s not too shallow, nor too deep. Perfect for an LLM to extract KG triplets from your data. Here it is as a thin StrEnum of the 5 node types:

In case you want more granularity over your POLE+O model, you can introduce subclasses, such as:

• Person: individual, alias, persona

• Organization: company, nonprofit, government, ...

• Location: address, city, region, country, ...

• Event: meeting, transaction, communication, ...

• Object: device, software, document, task, topic, project

To learn more, I have a full article on agentic GraphRAG and another one on designing an ontology for your context layer.

Version 2, the lighter one, is based on LLM wikis, such as Scrabble, my current agent memory. Markdown files with YAML frontmatter and cross-references, sitting directly on top of your existing infrastructure (Obsidian, Notion, Google Drive) — exactly how Scrabble runs over my own second brain. Google recently formalized this pattern as the Open Knowledge Format (OKF), a vendor-neutral directory of markdown + YAML whose only required field is type [3]. The filesystem is the state; no database required.

Which do you pick? It depends on how much control over the data model you want and your technical depth. Personally, I use both: Scrabble for everyday research across my second brain, and I plan to use Tree when I need precision at scale. Such as mining high-signal knowledge from 10,000+ documents.

Still, when it comes to implementing a unified memory layer on top of a knowledge graph, I keep getting asked one question: is a single database (MongoDB) enough? Why not throw in a specialized graph database like Neo4j or a vector database like Pinecone?

Is MongoDB Enough?

The simplest system that works wins. Instead of 3 databases (a SQL/NoSQL store, a vector DB, and a graph DB), you want one store that does all three. The payoff is concrete: far less operational overhead (one production DB, not three), less deep expertise required (being a power user of one engine beats being mediocre at three), and the ability to join documents, vectors, and graph in a single query. That last point matters most for memory: a single-store join is faster, cheaper, and lower-latency.

MongoDB is a good example, because it’s schema-less, supports all indexing operations, and has a huge ecosystem around it, with both an open-source version and their self-managed Atlas version. Another option I’ve tried that works well, but with more developer friction, is Postgres.

Also, as the cherry on top, you get lineage for free. Which is essential for adding references, understanding where each node came from, or simply preparing for an audit. You keep references on each knowledge-graph node instead of copying all data onto it: the document(s) a node was extracted from, the user who owns it, other metadata. That gives you easy lineage and versioning, and lets your memory connect to the rest of your system with no cross-database sync tax.

Now what about scale? For personal assistants, where you have only 100-10,000 documents, a single MongoDB cluster is more than enough. But even for medium to big enterprises, you can easily scale to millions of documents by adding more clusters (aka horizontal scaling).

After a conversation with a principal MongoDB engineer I understood that the real bottleneck is RAM. RAM is the most expensive component from your database cluster and you want to keep it as low as possible.

So index only what you query. Your memory holds 2 snapshots of the same knowledge: an append-only log (the immutable record of everything ingested) and a materialized view (the queryable graph rebuilt from that log). Vector indexes are inverted indexes ≥ the data they cover, so indexing both blows ~10 GB of data up toward ~40 GB of RAM.

Keep the log on disk (no vector index) and index only the materialized view, and it collapses back toward ~10 GB with the same data, at a quarter of the RAM.

When does it make sense to use a specialized graph DB like Neo4j? The reality is that for most use cases, you will do only 2-3 hops traversals. Such as getting all the preferences of a user, and the documents/conversations they were extracted from. In these cases, a single database that does it all is amazing. But there are cases when you should reach for a specialized graph DB like Neo4j. For example, when you need to do 3+ hop traversals, your whole business logic relies on graphs or simply as an internal visualization tool. You could sync your MongoDB production database to Neo4j, just for exploration reasons, as their Cypher engine + visualization ecosystem is stronger.

Using Your Context Layer With Any Agent

The wiki version is the simplest to switch between agents. If your context layer is shipped as an LLM wiki, switching harnesses is just handing the new harness a path. For this article, I literally pointed the harness at the research wiki folder from my Second Brain. Without any fancy skills or plugins in place.

Or a step further, is to point it at my entire second brain — that’s Scrabble — whose AGENTS.md explains how to navigate it and which CLI tools and skills to use. It works out of the box because it’s just files.

The MCP version is a bit more work, but not much. Serving memory over an MCP server means re-pointing one config entry from one harness to the next — switch from Claude Code to Codex by configuring the path to your MCP server, and your whole memory moves with almost zero friction, regardless of harness or model.

Each MCP server ships its own tools (read tools and write tools), and the tool descriptions are the contract the agent reads. It gets especially interesting on the write side, where the tools decide when to persist (e.g. parsing a whole conversation into memory when it ends).

Skills are where the leverage compounds. On top of this memory, you can write deep-research, writing, or coding-agent skills that remember your preferences. A few of mine, all part of Scrabble:

• /research, which runs deep research on a topic and produces a localized wiki (I have an open-source version in my latest ai-research-os-workshop talk made for the AI Engineer World’s Fair SF - full video);

• /article-guideline, which expands a brain dump into an article plan anchored in a wiki;

• /squid:plan, part of my Squid software factory, which plans a feature from my spec, codebase, and memory — injecting my personal takes on how I write software.

What’s Next

Agent memory is an unsolved, fast-moving topic. These designs work, but they’re far from perfect, and your data is still segregated across Obsidian, Notion, local files, and messages in ways that are hard to unify. In practice, I lean on the lighter wiki-based approach. One per project does the job. I reserve Tree’s heavy knowledge-graph memory for the few high-precision cases that earn its cost.

But here is what I’m wondering:

Your data can move between harnesses, but can your skills? I have this issue with Claude Code’s workflows and agents embedded into my skills.

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

Special thanks to MongoDB for sponsoring this article and keeping it free!

Explore Next

1. LangChain. (n.d.). Open Models, Open Runtime, Open Harness — Building Your Own AI Agent With LangChain and Nvidia. YouTube. https://youtube.com/watch?v=BEYEWw1Mkmw

2. Soria Parra, D. (n.d.). The Future of MCP. YouTube. https://www.youtube.com/watch?v=v3Fr2JR47KA

3. McVeety, S., & Hormati, A. (n.d.). Introducing the Open Knowledge Format. Google Cloud. https://cloud.google.com/blog/products/data-analytics/how-the-open-knowledge-format-can-improve-data-sharing

4. getzep. (n.d.). Graphiti. GitHub. https://github.com/getzep/graphiti

5. Neo4j Labs. (n.d.). agent-memory. GitHub. https://github.com/neo4j-labs/agent-memory

6. mem0ai. (n.d.). mem0. GitHub. https://github.com/mem0ai/mem0

7. topoteretes. (n.d.). cognee. GitHub. https://github.com/topoteretes/cognee

8. getzep. (n.d.). Zep. https://www.getzep.com/

9. HydraDB. (n.d.). HydraDB. https://hydradb.com/

10. Iusztin, P. (n.d.). ai-research-os-workshop. GitHub. https://github.com/iusztinpaul/ai-research-os-workshop

11. Iusztin, P. (n.d.). squid. GitHub. https://github.com/iusztinpaul/squid

Images

If not otherwise stated, all images are created by the author.

The scariest AI failures are the silent ones.

You ship a change to your agent on a branch — a new feature, a prompt fix, a quick refactor. No errors. No complaints. Everything looks fine. But does it still work, or did you quietly break something that worked yesterday?

As Alejandro Aboy puts it: “the fact that they’re not complaining doesn’t mean there’s no issue going on.” A quiet user is not a happy user. Usually, it’s the opposite.

The more Alejandro and I talked about AI evals and EDD, the more his struggles sounded like mine. A story from builders to builder.

1. You can break what already worked. Change a prompt, refactor a tool, and an old feature quietly regresses. Alejandro lived it: cleaning noisy instructions out of his agent’s system prompt made it start fabricating IDs it used to get right. You only catch that by running the same tests before and after the change, and comparing.

2. The feature is brand new, so you have nothing to test it on. No dataset, no historical traces, no ground truth. Yet you still need to know whether it works, and how well. So how do you generate realistic test data fast, then feed it to evaluators that turn it into hard performance numbers?

This is the case study that gives you a plan of attack for both: Evaluation-Driven Development (EDD)

How to prove a new feature works, and didn’t regress, before you merge. It comes from a recent conversation with , a senior data and AI engineer at Workpath who owns the entire data stack, built the Workpath AI Companion, and writes The Pipe and The Line Substack.

And we won’t keep it abstract. Every example comes from one real product: Workpath, a strategy-execution SaaS that keeps large companies’ OKRs and initiatives aligned. (OKRs — Objectives and Key Results — are the goal-setting framework teams use to name what they want to achieve and the measurable results that prove they’re getting there.) Its AI-native feature is the Workpath AI Companion: an agent that scans a company’s strategy and OKR data end-to-end to keep enterprise teams aligned. It’s the exact system Alejandro runs EDD on every day.

So when we say developing a new feature, picture shipping a change to that Companion and proving, before you merge, that it works and didn’t regress.

The Develop-a-Feature Workflow

Imagine. You start working on a new feature, branch, and develop the change. But before you merge, you have to answer 2 questions:

1. What’s the performance of my new feature?

2. Did my change introduce any regressions into the existing codebase?

Only when both look good you accept the pull request. EDD helps you answer those 2 questions.

Every feature is hypothesis-first. As Alejandro frames it, “I have a hypothesis... and everything should lie around that.” Every change starts as a stated hypothesis on a branch.

Based on that hypothesis, EDD runs a simulation and scores the results to answer the 2 questions.

Every feature ends in a PR, backed by an experiment, with clear traces and metrics. Framing the eval results as an experiment allows you to compare current results to previous ones, detecting regressions or tracking improvements.

This is how you can compare two experiments in Opik:

What about the process that happens between starting a new feature and its experiment?

From an architectural perspective, we have:

• The AI application, which can be an AI agent, workflow or a simple chatbot. In Alejandro’s case, it’s an AI agent built with Agno. More precisely, it’s the Workpath AI Companion he is building. But due to data privacy reasons, during the demo, he could share only a mock of the data.

• A headless evaluation harness, powered by Claude Code.

• An AI observability and evaluation platform responsible for capturing traces, managing eval datasets and evaluators, running experiments and comparing results. Alejandro is using Opik. The tool is open-source, but for ease of use, you can also try out their managed platform for free here for 25k spans/month.

Now... how do we generate data for these experiments? How do we get the traces? How do we populate the evaluation harness with the right context? We will see how all of that falls into place, where everything starts with two modes.

Two Modes: Manual Quick Check vs. Automated Experiments

The two modes are modeled by the /edd skill, which has two inputs: Mode and Aggression.

Mode 1 is a quick, manual check. You fire around 30 fresh traces, let Claude Code read them back from Opik one by one, and trigger a judge by hand only if you want a score. As Alejandro describes it, “it won’t trigger automatic evaluations; you trigger them manually.” No dataset, no experiment, ephemeral, minutes. His favorite for a small change: his Substack Author Agent kept over-asking for the publication URL on every new trace. A tiny, targeted fix, exactly what Mode 1 is for.

Mode 2 automates the judgment. When you touch a lot or ship new functionality, you turn the traces into a dataset and run an experiment, both Opik objects, where the judges score every item automatically and produce an experiment you can compare across runs. This is the only way to catch a subtle regression, because you compare 2 experiments.

Both modes start from a hypothesis on a branch, emit fresh simulated traces, and can use the same evaluators. The mode only changes whether the evaluation is done by hand or automatically.

The Aggression setting controls how adversarial the simulated traces get, from happy-path up to fully adversarial. As you turn up the knob, simulated traces get more aggressive, finding harder and harder corner cases to break the agent.

The secret sauce of Alejandro’s EDD approach is in how he uses Claude Code to simulate fresh traces.

Scope the Change and Simulate Its Traces

Remember. Our goal is to simulate relevant traces to test the performance of our feature. To do that, we use Claude Code to read the agent’s source code, especially the code around the new feature. After, we retrieve old traces (stored in Opik) that are relevant to our current code.

Based on these two signals, we generate ~30 traces targeting the new feature’s functionality.

The traces need to be high signal and as diverse as possible. The goal is to find holes within our system and fix them, not to validate what currently works.

To achieve that, the traces are generated based on 2 dimensions:

• Regression evals (what worked still works) vs. capability evals (can do it on new things

• Happy path (easy: testing the core logic) vs. adversarial (hard: finding edge cases, such as missing data, faulty tool descriptions or guardrails)

During generation, we can configure these parameters. For example, if we go full adversarial, the probability of finding errors increases. Which isn't necessarily a good thing, as you don’t want to overoptimize in advance either. You want to make the system as good as possible on the hot path. You don’t want to waste time on scenarios that might never happen. That’s why anchoring your trace generation to existing traces is an essential step for properly understanding the user’s behavior and which components to target when generating the traces.

⚠️ Important! Even if we simulate the data, we still want REAL traces and outputs to evaluate on.

This is what we have to do. The pipeline starts from the data, not from invented inputs. Claude Code analyzes the current traces to learn what inputs are worth generating, so we simulate only the inputs, NOT the outputs or internal state.

That’s the whole point. Synthesize the outputs too and you hit Alejandro’s problem: “every time I try synthetic datasets, I was losing everything the agent was doing beyond the response.” Grade the final answer alone and a wrong tool call stays invisible.

To get there, we send each simulated input to a headless copy of the agent, which runs for real: selecting tools, calling the staging backend, handling whatever comes back. As it runs, Agno records the full tool-call history and outputs into its OpenTelemetry trace, and the agent emits it to Opik.

Using this strategy, we simulate the inputs, run the agent, and record the trace with real values produced by the agent.

A simulated trace is only as trustworthy as the state the agent was in when it ran, and recreating that state is the hardest part.

Context Population: Mocking Production State

The hardest part of agentic evals is getting the agent into the right state, so it passes or fails for reasons that actually relate to your hypothesis. A trace generated from the wrong state is a useless trace.

In Alejandro’s use case, roughly 90% of the agent’s tools are API calls, so Claude Code gets a token and hits the real internal backend through a staging mocked account that already holds data. For the happy path, it pulls real goals, OKRs, and teams. To go adversarial, it forces errors and asks for data that doesn’t exist.

The reusable trick is where the context gets injected. Before the agent boots, Claude Code calls the API and injects the user’s context into dedicated system-prompt sections. So the agent greets you with “Hi Paul, want to check goals from the coding AI team?” It runs “as if for real.”

Alejandro is explicit that this is not pytest-style fixtures, but “the prompt is the only thing the LLM sees.” A faithful prompt-level state is a faithful enough production proxy. You mock at the system-prompt layer and stop worrying about reproducing the whole backend.

So instead of using the standard way of using fixtures to populate the backend, you can bypass everything and directly inject the context into the system prompt. From the LLM’s perspective, it’s the same thing.

The last step is to transform the traces into an evals dataset.

On-Demand Datasets

You want two types of eval datasets:

1. A persistent, hand-built evaluation set that tests the core business logic. Useful for catching regressions.

2. An on-demand, synthetic dataset used to evaluate the feature you are working on.

We are interested here in the second option.

Via the /edd skill Claude Code assembles an Opik dataset, on the fly, from the branch-tagged simulated traces. The dataset is tagged so a later run can filter straight to it, then kept or thrown away.

Before committing to a big sample, Alejandro fires a couple of runs as smoke checks, “to catch anything awful” before spending tokens. Then he checks that the dataset’s coverage is good enough to be worth running an experiment against. Small, cheap, and it saves the expensive mistake.

Because the dataset is cheap to regenerate and scoped to one change, it’s disposable. Optionally, you might promote a couple of high signal traces into the persistent regression set.

A dataset is only useful once you’ve decided what metrics to use — aka the judges.

Define the Judge

You want to support 2 evaluator types.

Code metrics score the structural things deterministically, server-side, free, no LLM, like whether it called the tool or whether the format is right. Always try to evaluate a given metric via code metrics if possible.

LLM judges score the subjective things, like completeness, accuracy, and ranking quality.

Both evaluators are designed as binary classifiers: verified, or not. The urge to introduce 1-5 likert scales is huge. But the thing is that it’s incredibly difficult to get it right. What’s the difference between 2 and 3 or 4 and 5? Even when using multiple human annotators, the labels are inconsistent. With binary labels, the decision is clear: it’s correct or not. Which makes it incredibly easy for the LLM to get it right.

To get some nuance, on top of the binary labels, you want to add a critique that explains in 2-3 sentences why the output is correct or not.

The judge model is deliberately a different model than the agent, so the two don’t share blind spots and the judge can’t rubber-stamp its own family’s mistakes.

But here is the trick! The evaluators are static, carefully defined and calibrated up front from the codebase. The loop regenerates traces and datasets, not the metrics. When implementing LLM judges, it’s extremely important to align them with the domain expert. Once they are working well, you can use them for inference, which we are doing here on the dynamic datasets.

The judges are configured within Opik, using their API to call the model to score each sample from the dataset.

In the image below, you can see the evaluators Alejandro configured for each experiment:

• Content Completeness

• Metric Accuracy

• Ranking Quality

• Relative Grounding

• Response Directness

• Semantic Search Accuracy

• Skill Selection

With the dataset built and the judges defined, you run the experiment. And you run it twice.

Run and Compare Experiments

The experiment runs in Opik, scoring the dataset against all the judges to produce a score distribution.

In a feature Alejandro was working on, he cleaned all the noisy instructions out of his agent’s system prompt, “hygiene before” vs. “hygiene after,” and ran 2 experiments over the same scope in Opik’s comparison view.

The after had regressed on one judge: tool-call parameter inference. The agent should remember which ID to pass to a tool, but the cleanup made it “get lost and fabricate IDs.” EDD caught his own change before it shipped.

Comparison matters because failure hides where a single trace can’t show it. Trace-level evals are usually fine, but problems surface across 5, 10, or 20-message conversations. Around the 10th message, the model slides into a “context rot zone”: a request that earlier earned a cooperative “let’s work with this” now gets “what do you mean by that?”

A user asks the agent to “scan 50 teams, get me all the OKRs.” It pushes back and offers to go progressively, returning 17 copy-pasteable tables. But by trace 21 of a 20-message conversation, you’re at 200k total tokens, paying heavily without caching.

These are the kinds of errors proper evals protect you from! Not only performance, but also latency and cost issues that can blow up your infrastructure overnight.

Don’t Run Online Evals

Everything so far ran offline, on a branch, before the merge. The expensive default everyone reaches for instead is always-on online evals. That’s the trap.

Alejandro made the same mistake.

He thought running online evaluations on all the traces was essential for production. Until the bill! It was on credits, not cash, but it would have been around $2k a month just from triggering a few evaluations: “the bill just pops in — in one second you have thousands of dollars in debt.”

So you recalibrate. What’s the actual risk of evaluating whether the agent leaked an ID to the user? Low. So you sample or look for a pattern, run the heavy judges offline, and cap spend first: “consume an amount you know you can afford.”

The good news? The whole /edd skill and headless harness implemented by Alejandro via Opik is now an installable open-source Claude Code plugin. You can also create a free account on Opik with 25k spans/month to try out this EDD strategy on your own project.

Final Thoughts

You’re already driving one AI process with another. Would you hand the whole thing over to an agent that reads the traces, understands the agent’s mistakes, gets the signal from the evaluators and writes the code changes itself?
— Paul

Alejandro would want exactly that, on one condition. He’s tried prompt-only optimizers and doesn’t trust them, because they change the prompt but never test the agent’s full harness. Until then, the human stays in the loop, and every change earns its pull request.

Opik has been shipping exactly that: Test Suites, Agent Configuration and Playground. Where does your hand-rolled loop go from here?
— Paul

Because they tackle the prompt-only-optimization gap, Alejandro is bullish on adopting them — he expects they’ll close the whole agentic loop: analyze the code & failures → generate inputs → call the agent → build a dataset → evaluate each sample → fix the code → repeat, all in one place.

Try out Alejandro’s EDD code on GitHub, while leveraging Opik’s free tier as the observability platform.

But here is what I’m wondering:

Where do you draw the line between online and offline evals? What do you actually run always-on in production, and how do you cap the spending?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

Thanks again to Opik for sponsoring this case study and keeping it free!

If you want to monitor, evaluate and optimize your AI workflows and agents:

Try Opik for free

Images

If not otherwise stated, all images are created by the author.

and I were planning our upcoming book when we kept hitting the same realization. Just as LLMs got commoditized, the harness around them is commoditizing too, hardening into a handful of standardized, batteries-included frameworks. Once it’s a commodity, the hard question flips. It stops being “how do I get an agent running” and becomes “for each piece, do I build, configure, or just use it?” And that line is blurry.

Overbuild, and you burn weeks reimplementing a tool loop, a permission system, and a sandbox that is already available for free. Under-build, and you lean on the defaults forever, never building the one layer that’s actually yours, your context layer, your moat, so you stay a renter of someone else’s system.

The harnesses fall along a spectrum, from tool-like ones you customize as a user (Claude Code, Codex, OpenCode) through framework-type ones you build with, like Pydantic AI in Python or pi in TypeScript.

This article hands you that system design: the ~80% blueprint that’s conceptually the same across Claude Code, OpenCode, Codex, and pi, walked component by component, with one conclusion per piece.

We start with the big-picture architecture, the shape almost every harness shares, then walk it component by component: the tools the model calls, the catalog of agents, how subagents spawn and stay contained, how skills load cheaply, where memory really lives, how sandboxes both protect and scale you, and the permission layer with almost no AI in it. Each one closes on a verdict that fills in the map.

Build the Layer That’s Actually Yours (Product)

This article shows the system design of a harness, the commoditized part. The real value is the business layer you build on top of it. That’s what my Agent AI Engineering course teaches, built with Towards AI: only what you need to deliver value, not how to rebuild the harness.

35 lessons. 3 end-to-end portfolio projects. A certificate. And a Discord community with direct access to industry experts and me.

Built for software, data engineers or scientists transitioning into AI engineering.

Rated 5/5 by 300+ students. The first 7 lessons are free:

Start here

The 80% Every Harness Shares

Roughly 80% of a harness’s blueprint is identical regardless of the tool or framework, because every harness solves the same problems with different techniques. That shared 80% is exactly the part you mostly use as-is. The decisions about what to configure or build live in the remaining slice. At the highest level: a user message goes in, an answer comes out, and everything between is the harness.

In between, here are the 5 core layers:

The Agent is the innermost piece: the agentic loop (the ReAct loop, where the model reasons then acts) wrapping an LLM plus its tools. The LLM can be closed (Gemini, Anthropic, OpenAI) or open-source served over the OpenAI protocol on Modal, RunPod, GCP, or TogetherAI, or run locally with Ollama or llama.cpp. This loop is the core: strip away compaction, task budgets, and thinking and it’s roughly 150 lines.

The Harness is everything wrapped around the agent: a message queue with a priority gate, the sandbox, hooks, services (LLM gateway, memory, LSP servers, MCP client), skills, the permission system, an agents catalog, and subagents, with context engineering sprinkled everywhere.

The Runtime is the durable execution layer the whole harness runs inside: Prefect, Temporal, Kitaru. It gives you non-blocking human-in-the-loop, scheduling, durability and caching, and a credentials proxy.

The Presentation layer is how you talk to the harness, whether TUI, web, mobile, WhatsApp, or Telegram. The interesting question is how the same agent serves many front-ends, and there are 2 real patterns. One is a pub/sub bus, OpenCode style, where a headless server streams events to TUI, web, and desktop clients over HTTP+SSE so many clients observe 1 live session. The other is custom services bridging into 1 in-process loop, the Claude Code style.

The Observability layer is tracing, logging, metrics and evals sitting across everything, with tools like Opik, Langfuse, or Braintrust.

To see how the components connect, let’s follow a single message down the happy path: user → TUI → message queue → wait for the agent to be free → agent → LLM → tool → LLM → tool → … → LLM → answer → TUI → user.

Meanwhile, The TUI sends and receives over SSE. The priority gate decides when to inject a new message between loops rather than interrupting mid-loop. When the context window nears its limit (tokens ≥ contextWindow − reserve), compaction runs, keeping the window as [system prompt] + [summary] + [recent tail]. And a single tool call can fan out into hooks, sandboxes, services, and permission gates.

This skeleton, the loop, the queue, the runtime wiring, the message journey, is the commoditized 80%. As there is a ton going on on top of the basic agentic loop, let’s explore all the core components of the harness to get an intuition on what can be customized or built on top of it.

The Tools

The set of tools the LLM can call inside the agentic loop is the most visible part of a harness. Everything the model can invoke conforms to a single shape, a name, an input schema, and an execute method behind a flat registry.

Ground that in what a real harness ships. Claude Code organizes ~40 built-ins into 10 families, and these are what you get for free:

• File I/O: FileRead, FileWrite, FileEdit, Glob, Grep. The model’s hands on your files. Read one, write a new one, edit in place, find files by name pattern with Glob, and search their contents with Grep.

• Execution: Bash. A single tool to run shell commands. The most powerful tool available that allows an agent to run shell, Python, TypeScript or in general interact with your machine.

• Orchestration: EnterPlanMode / ExitPlanMode, Sleep, Agent (spawn a subagent), EnterWorktree / ExitWorktree. These shape the work itself. Plan mode gates edits behind a read-only planning pass, Sleep pauses the loop, Agent spawns a subagent, and the worktree pair carves out an isolated branch to edit without touching the main tree.

• Tasks: TaskCreate / Update / Get / List / Output / Stop. A task state machine that lets the agent track a to-do list and run long jobs that outlive a single turn.

• Web: WebSearch, WebFetch. The window outward. Search the web, then pull a specific URL’s contents into context.

• MCP: an MCP tool factory + ToolSearch, ListMcpResources, ReadMcpResource, MCPAuth. This is how external tool servers plug in. The factory mints 1 tool per each tool from the MCP server to flatten out the tool discovery logic into a single tool set (e.g., /mcp__brown__edit_content_prompt). ToolSearch surfaces the right one when hundreds are attached, and the rest list resources, read them, and handle auth.

• Scheduling and misc: ScheduleCron, RemoteTrigger, Skill (a dispatcher, 1 tool with N skills by argument), LSP, AskUser. The odds and ends. Schedule a run on a cron, trigger one remotely, dispatch a skill by argument, query a language server with LSP, and hand a question back to the human with AskUser.

The built-in tool families are the commoditized surface. The customizer’s job is to configure which tools each agent may call. The architect’s is to build new domain tools as MCP servers plugged into the same registry. That’s where your product’s actual capabilities live.

Tools are what the model can do. The agents’ catalog is who does it.

The Agent Catalog Is Just a Config File

Each harness ships a set of predefined agents, and the version worth copying defines them as config, not code, because that makes them discoverable and pluggable without touching the loop. The format varies: a markdown file with YAML frontmatter, where the body is the prompt, in pi and Claude Code, or plain YAML or JSON in OpenCode. I reach for YAML, but the point is the same. The core fields are small: name, mode, model, tools, disallowedTools, and permission.

When you open Claude Code, you chat directly with the primary agent, the primary process. The trickiest part to understand is that both the primary and subagents can wear multiple hats. The agent catalog distcribes these hats, these modes the agent can take on.

A catalog worth copying looks like this, synthesized across the references (the mode: primary | subagent | all axis is OpenCode’s, while Plan, Explore, and General Purpose ship in Claude Code):

• Build (mode: primary), the default agent.

• Plan (mode: primary), read-only.

• General Purpose (mode: subagent), the fallback when no specific agent fits.

• Explore (mode: subagent), read-only search and locate, running on a cheap model.

• Code Reviewer (mode: subagent), read-only and git/diff-aware.

The code-reviewer subagent (in Claude Code) tools allowlist grants FileRead, Grep, Glob, and Bash(git *), while its disallowedTools denylist blocks FileEdit, FileWrite, and Bash(rm *). So it can read the tree and run git, but it can never edit a file or shell out to delete one. That dual allowlist/denylist, with rule syntax like Bash(git *). The safety trick is that the scope is narrowing only. OpenCode enforces it by deriving a child’s permissions from its parent’s, so a delegated agent can never out-permission the one that spawned it.

Use the bundled agents as-is for everyday work, then author your own as YAML or markdown files. You rarely need to build a custom agent. That usually happens when you build your custom application. For example, I did it only for my deep research and writing skills, which required a ton of customization. Ultimately, ending up as completely apps covered as a skill.

To get the full picture, let’s understand how subagents work.

A Subagent Is a New Loop

Most harnesses support subagents, though some, like pi, do it via plugins instead of natively. The hard part is keeping orchestrator and child communicating without the child’s full context polluting the parent loop and ensuring the orchestrator, “orchestrates” the subagents as expected. Remember that the orchestrator is an agent, not a workflow encoded in code, which means it can easily go off track and forget a step.

In Claude Code a subagent is not new code. It’s the same loop re-entered with a cloned context and a restricted tool list, and only a condensed summary flows back. A periodic ~30-second summarizer fork produces a live progress label plus a bounded final summary. The lesson to steal: a subagent is your existing loop, narrowed, with a summary on the return path. Only recently they started introducing subagents as new processes.

Harnesses almost never support swarm architectures where every agent talks to every other. They support a master–slave orchestrator topology where one main agent tracks the children.

Spawning is half the problem. The parent and child still have to talk, and there are three channels, ordered by how far apart the two run.

Cheapest is in-process: the child is a nested call, so its output is just a return value handed back to the caller. A queue sits one step out. The parent drops work on a message queue, the child consumes it, and the parent awaits a result event. Because the queue is a shared bus, other clients can watch the same exchange live, the way OpenCode streams a subagent’s events to many observers. Most decoupled are shared JSON files: a lock-serialized mailbox, one file per recipient, that agents in separate processes or worktrees write and poll. pi’s one-way subprocess, streaming JSON lines back to the parent, is the same idea narrowed to a pipe.

For most builders this is use-as-is, lightly configured. The spawn mechanism and orchestrator topology come standard, and all you configure is each subagent’s tool and permission scope and which agent it spawns. You only build when you need exotic isolation, like pi’s out-of-process model for untrusted children. That’s a rare need.

Now let’s see how skills fit into the picture.

Skills

A skill is one of the simplest implementations to understand yet one of the highest-impact things in the whole harness. It’s essentially a markdown recipe, instructions plus an allowed-tool set, that the model pulls in on demand.

Concretely, skills come from 3 sources merged together: bundled skills shipped with the harness (e.g. src/skills/bundled), defined skills dropped into .agents/skills, and MCP server prompts. The pipeline is short. A GetSkills step collects all 3 sources, caps the total at ~1% of the context window, assembles a single skills context, and wraps it as a <system_reminder>.

That 1% cap is the whole trick, and it works because of progressive disclosure. Skills are surfaced by name and description only, so the agent sees a cheap menu of capabilities and reads a skill’s body on demand, which is why the always-loaded skills context can be hard-capped at ~1% and still scale to dozens of skills. pi takes the same spirit further, surfacing its skills via prompt injection rather than as tools.

This is pure configure, or really authoring, and it’s the single best return on effort for the user and customizer tiers. Writing a markdown skill is the cheapest way to teach the harness a new workflow, and the 1% cap means you can pile on dozens.

Skills, tools, and subagents all hang off the loop, and they’re mostly things you configure. Memory is different. It’s the one component where you actually build your own layer.

Memory Is the Layer You Actually Build

In most harnesses, out-of-the-box memory is loaded directly into context, not via a tool. The model never calls a tool to “remember.” Relevant memories are read off disk and prepended to the system prompt before the turn runs, and new memories are extracted after the turn by a separate process.

A file-backed design, Claude Code-style, is worth grounding concretely. The store splits into 2 kinds of files. User-defined .md files come first: AGENTS.md is always loaded, and **/AGENTS.md is loaded dynamically per directory, on demand. LLM-extracted .md files come second: MEMORY.md is an always-loaded index, hard-capped at ~200 lines / 25 KB, while logs/YYYY-MM-DD.md is an append-only daily log where only the relevant logs are loaded. A small-model side-query ranks topic files from the log by their frontmatter description, not embeddings, and picks the top few to inject, which is debuggable and needs no vector store.

By default a forked extractor updates MEMORY.md live after each turn. A daily-log variant runs a nightly /dream distillation instead: a small LLM extracts the conversation into logs/YYYY-MM-DD.md, then a second distills those logs into MEMORY.md. In other words the pipeline looks like this: raw conversation → daily logs → durable memory.

Most harnesses use a file-based system for memory. Which is good enough for uses cases such as coding. Other tools, like Cursor or OpenClaw, build a vector index over your memory instead. That’s why many people report better memory from OpenClaw. As instead of parsing your whole memory as append only logs or forgetting context when building the MEMORY.md index, OpenClaw builds a vector index over your memory.

Here’s the heart of the build/configure/use thread, though. The defaults get you started and AGENTS.md is worth configuring, but the highest-leverage move is a custom memory layer behind an MCP server, a database exposed through an MCP server with your own read/write logic. Because it’s harness-independent, you jump from Claude Code to Cursor to anything and the agent instantly picks up who you are.

Real independence means owning your own context layer.

This is the one place to build, and it pays off for every tier that’s serious. The context layer behind an MCP server is the moat. It’s harness-portable, fully yours, and the thing that makes the assistant your assistant.

Owning your context is about what the agent knows. The next layer is about where its code runs, sandboxing, which protects you and, surprisingly, lets 1 harness scale to many jobs.

The Sandbox: One Jail, Many Remote Workers

The obvious reason for a sandbox comes first: it keeps the agent in a controlled environment with no direct access to your machine. Establish the key separation early.

Sandboxing lives at the Bash and PowerShell tool layer, not the UI. When the model issues a Bash tool call, a decision runs about where it executes. If remote, the command runs in a sandbox such as Modal. If local, the harness asks whether to use a sandbox at all: yes means it runs inside a local sandbox (Docker, Firecracker, …). No means it runs directly on your machine.

The enforcement detail worth stealing, the way Claude Code does it, is that the jail is derived from the same permission rules the agent already uses, and it always denies writes to its own settings file.

On top of security sandboxes can change how we define software architecture. Reframe sandboxes as workers from classic distributed systems: each sandbox is a worker that runs jobs in parallel, and 1 harness can manage and scale many of them. So the same harness that protects you locally can fan out dozens of remote jobs. Depending on your sandbox type, you can run data ingestion jobs or even training jobs if the VM has a GPU. Everything from your harness. Codex is a harness that is all in on remote sandboxing.

Now, let’s wrap up the article with the most important component: the permission layer.

The Permission Layer Has Almost No AI in It

The permission system is the hardest part to reason about, and the strange thing is it has essentially no AI in it, yet it’s what makes the whole system safe to run. Its job is narrow: for every tool call, decide to (a) run it, (b) ask the user, or (c) deny it.

The structure has 2 flavors. Agent modes change default behavior: default, acceptEdits, bypassPermissions, and plan. User-defined rules live in config, in .agents/settings.json and .agents/settings.local.json, where you declare what the agent can and cannot run, including wildcard rules like Bash(git *). The harness combines mode metadata and user rules at runtime to resolve each call.

The “Can use tool?” question has 3 outcomes.

Allow calls the tool. Ask surfaces it to the user, and on allow it calls the tool, while on deny it synthesizes a denial tool-result and continues. Deny synthesizes the denial directly.

When deciding what to do, the harness runs tool filter → user settings → mode.

Here’s the counterintuitive payoff. “Bypass everything” is not total. Plan mode is enforced prompt-side, via a system reminder telling the model to only edit the plan file.

Which shows how fragile these mechanisms still are, as we just hope for the best that the model will pick up the instruction.

You almost never build this. But it’s incredebly important to properly configure it. It’s probably the most important part to configure right to ensure it has just enough access to your data and machine.

What’s Next

These are just the core components that almost any agentic harness needs and has.

But there is more to it.

Worktrees for parallel isolated edits, multiprocessing subagents for true parallelism, and a plugin system for extending the harness without forking it. Which I will address in future articles.

But here is what I’m wondering:

Which component did you decide to build rather than configure? Was owning it worth it, or did you reinvent something the harness already had?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

Images

If not otherwise stated, all images are created by the author.

Two months ago, I started building unified memory layers on top of knowledge graphs. One question kept coming back from readers. How do you handle entity resolution and deduplication without corrupting the graph?

Rather than guessing, I spent serious time studying how mem0, cognee, and Neo4j actually solve it. The recurring question exposes a confusion almost everyone shares. People treat entity resolution and deduplication as the same step.

That confusion is exactly what corrupts graphs. People collapse naming and identity into 1 fuzzy check.

Also, if the merging step is not properly designed, 2 different real-world entities can silently merge, corrupting your graph.

Resulting in losing the trust in your graph that made it worth building. The graph quietly rots. Nobody trusts it, and the entire memory layer you invested in goes unused.

The failure is invisible until it becomes expensive to undo. The fix is to separate naming from identity.

We will walk through the end-to-end pipeline. This includes LLM extraction, entity resolution for naming, embedding the full node and deduplication for identity. Plus, the 2 safety nets most tutorials skip.

We covered the full memory-system design and the ontology design in prior articles. This piece focuses only on keeping the graph clean. By the end, you will be able to design a graph that stays clean and usable as it grows.

Why We Killed RAG in Production (Product)

This article shows how to keep a graph memory layer clean. In a recent podcast, I covered the decision that comes before it: whether you need retrieval at all.

I explain why we killed RAG for a financial advisor product. All of an advisor’s data summed to 64,000 tokens, so loading the full context beat RAG’s zigzag retrieval loop. The formula I use: your data-to-context-window ratio.

We also get into regretting MCP everywhere, treating vibe-coded output as a compilation step, and why AI evals become the real job once the model writes the code.

Watch the episode

One Pipeline, Five Steps

In goes a document or a conversation turn. Out comes a set of canonical, deduplicated nodes correctly wired into the existing graph. Everything between is about making sure each new node is named and identified right.

First, an LLM extractor reads the text and emits entities and relationships connected by (entity, relationship, entity) triplets. It anchors within the POLE+O, Facts, and Preferences ontology. This ensures it only extracts the entity types you actually care about, as we explained in depth in this article.

For example, a sentence about a person working at a company becomes a (Person)-[:WORKS_AT]->(Organization) triplet. The ontology told the extractor those are the types that matter.

If using only LLMs for extraction becomes too costly, you can use a cost-tiered cascade here, starting with fast statistical models like spaCy for common entities, moving to zero-shot models like GLiNER for domain-specific types, and falling back to an LLM for complex cases.

Before touching the graph, we must decide what this new entity should be called. The system normalizes its name against existing nodes of the same type. This is the finding-the-canonical-name step, and no merges happen yet.

Next, we compute an embedding over the entity’s full context. This includes its name, type, and attributes. We embed more than just its bare name. This is what later lets deduplication compare identity rather than spelling.

We compare the embedded node against existing nodes. This decides whether it is the same real-world entity as one already in the graph.

Based on the deduplication outcome, the system makes a final routing decision. It either merges into an existing node, flags the pair for human review, or adds a brand-new node.

A new mention of a company gets extracted as a typed entity. The resolution step normalizes it to a canonical name. Then, it gets embedded with its full context so we capture its semantic meaning. It is compared against existing same-type nodes to verify its identity. Finally, it gets added, merged, or flagged for review.

Resolution and deduplication are 2 distinct decisions doing 2 distinct jobs. Let’s zoom in on each one.

Entity Resolution: “What Should We Call This?”

During resolution we find the canonical name for each entity. It answers “what should we call this?”.

It handles typos, acronyms, and surface-form similarity. These are the noisy ways humans and documents write the same thing. It uses exact, fuzzy, and semantic matching in a short-circuit chain.

The short-circuit chain passes the entity to the next matcher only if no confident match is found. If exact match fails, it tries fuzzy match. If fuzzy match fails, it tries semantic match (using light embeddings only on the name).

But it matches only against the names of existing nodes of the same type. You never compare a PERSON name against an ORGANIZATION name.

“NYC” resolves to “New York City”. “JP Morgan” resolves to “JPMorgan Chase”. The 3 forms "John Smith ", "john smith", and "Jon Smith" all collapse to 1 canonical “John Smith”.

This happens because resolution absorbs whitespace, casing, and typo variations. Fuzzy string matching uses token-based comparison to handle word order and partial matching for abbreviations. At this stage the system only updates the node’s canonical_name property. No graph merges happen yet.

Often, you also keep track of a list of aliases for each node. Whenever you find a new hit via fuzzy or semantic match that doesn’t match the current canonical_name, you add it to the list of aliases. Like this, in future checks you can speed up matching by checking the alias list first.

Similar names are not strong enough evidence that 2 entities are identical. This is the line most people blur. Blurring it is what causes silent corruption.

Apple the company is not Apple the fruit. They have different types, so type-gating already separates them. A harder example is Jensen Huang the CEO of NVIDIA versus a doctor in Taipei with the same name.

They have the same name and the same type. Yet they are 2 different real-world people. Naming similarity alone would happily fuse them.

Still, canonical names are extremely useful for GROUP BY operations where, during querying and visualizations, we can quickly understand the data. During human review, we can even spot duplicates and resolve them manually.

That is why identity is a separate decision. Resolution has told us what to call the node. It has deliberately not told us whether the node is a duplicate.

That second, riskier question belongs to deduplication.

Deduplication: “Is This the Same Entity?”

Deduplication is the identity layer. It answers the harder question: “is this the same real-world entity?”. It is the step where merges actually happen [5].

In goes 1 embedded node. Out comes a single routing decision: merge into an existing node, flag it for review, or create a new node.

The system embeds the full entity context. It compares it against existing nodes using semantic and fuzzy similarity across that full context. The richer signal is what lets it distinguish 2 same-named, same-type entities that resolution could not.

By the context of a node, we refer to the entity’s attributes such as its text, image, video content or even its metadata properties such as a person’s email or date of birth. Or an object’s model or manufacturer. Still, you don’t want to embed everything, such as identifier, but per each ontology type pick the fields that contain the highest signal.

The combined deduplication score is an explicit weighted blend. It uses the embedding score multiplied by 0.7 and the fuzzy score multiplied by 0.3. Based on a similarity score from 0 to 1, we have 3 bands.

High confidence (≥0.95) triggers an auto-merge. Medium confidence (0.85–0.95) flags the pair for human review. Low confidence (<0.85) creates a new node.

Near-certain identity is allowed to merge automatically. The uncertain middle is escalated. Weak evidence just becomes a fresh node.

False merges silently corrupt the graph. The corruption is invisible until it is expensive. Take the Paris example: 2 LOCATION nodes both named “Paris”.

One is the capital of France, and the other is Paris, Texas. They have the same name, the same type, and very similar bare-name embeddings. But they are 2 different places.

The dangerous part is the middle band, the gray area. This is where the system is not sure and a human has to step in.

When Confidence Lands in the Gray Zone

When a deduplication score lands in the medium band (0.85–0.95), the system deliberately does not merge. It flags the pair for a human to decide, as merging is a dangerous operation we should be really deliberate about.

The source node gets tombstoned, meaning it is kept queryable for forensics but skipped from future matching. Actually undoing a merge means re-ingesting the source data. That reversibility cost is the whole reason for the gray zone.

Whenever a new entity is flagged for human review, a new node is created and a (:Entity)-[:SAME_AS {status:'pending', confidence}]->(:Entity) edge is added inside the graph itself. The human review step transitions that status to confirmed or rejected. The review queue is just a Cypher query over pending SAME_AS edges, ordered by confidence.

For each flagged pair, the reviewer answers 1 question. Is this actually a duplicate, a new node, or neither?

This usually happens to entities that are related but not identical. The Codex model and the Codex CLI are related, but not the same object. The same applies to Jensen Huang the CEO versus a same-named doctor in Taipei.

This is hardest at the start of an entity’s lifecycle. When metadata is scarce, similarity spikes, and you risk polluting 1 node with another’s attributes.

Human review catches the uncertain pairs the live pipeline surfaces. But some duplicates never get surfaced at all. That is the gap the dream pipeline closes.

Cleaning the Graph While It Sleeps

While the system ingests documents, data often flows through in parallel. If 2 entities are processed at the same time, the resolution and deduplication steps never get to compare them against each other.

The system would never check whether Claude Code from Conversation X and Claude Code from Document Y are the same entity, because neither existed in the graph when the other was written.

You run a dream pass every night. It re-runs the deduplication pass on recently ingested nodes only. Otherwise, you will have to loop through all nodes in the graph. Which as the graph grows, becomes increasingly expensive.

It does not run the full resolution chain. Because the embeddings were already computed at ingest time, this is a light operation. It is primarily database reads and writes, not fresh model calls. Since it mostly adds I/O pressure, run it when organic traffic is low, which is usually during the night, hence the name the dream pipeline.

What’s Next

I’ve spent the past 4 months building unified memory layers on top of knowledge graphs, and I learned that keeping them clean is the hardest part. Keeping your knowledge graph clean is the maintenance step that decides whether the graph ever gets used. A graph full of noise, fragments, and false merges will not be trusted or queried.

In case you want to learn more, remember that we also covered the full memory-system architecture via knowledge graphs and the ontology design in prior articles.

But here is what I’m wondering:

What are the core strategies you’ve used to keep your knowledge graph clean and usable? Something close to our approach here, or something completely different?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

References

1. Iusztin, P. (n.d.). Understanding the Neo4j Graph Agent Memory System. Decoding AI Magazine. https://www.decodingai.com/p/understanding-neo4j-graph-agent-memory-system

2. Iusztin, P. (n.d.). Ship a Knowledge Graph Ontology in 5 Minutes. Decoding AI Magazine. https://www.decodingai.com/p/ship-a-knowledge-graph-ontology-in-5-minutes

3. POLE+O Data Model. (n.d.). Neo4j Labs. https://neo4j.com/labs/agent-memory/explanation/poleo-model/

4. How Entity Extraction Works. (n.d.). Neo4j Labs. https://neo4j.com/labs/agent-memory/explanation/extraction-pipeline/

5. Entity Resolution and Deduplication. (n.d.). Neo4j Labs. https://neo4j.com/labs/agent-memory/explanation/resolution-deduplication/

Images

If not otherwise stated, all images are created by the author.

For a while now I’ve been trying to build a proper memory layer on top of my research, writing, and content creation. Today it all lives in my Second Brain in Obsidian, where the primitives are files like notes, videos, and articles.

What I actually want is to shift those primitives from files to entities and relationships, such as people, locations, objects, topics, preferences, and facts. I want the memory to get closer to reality so I can watch how things evolve over time. I want a knowledge graph.

Everyone agrees knowledge graphs and GraphRAG provide a more performant substrate for a unified agent memory layer than plain RAG. But kicking one off is far harder. The resistance always collapses to the same wall: how you model your data. Your ontology is the hardest part of the system.

If you can’t define your ontology properly for your domain, the graph won’t represent the reality you want. The right entities and relationships simply aren’t there. As a result, GraphRAG ends up performing worse than the simple RAG you were trying to beat.

This translates straight to a memory layer. There’s no dodging it. Even if you stay file-only (a “virtual knowledge graph,” like an LLM knowledge base over your notes), you still hit the same data-modelling question: which primitives, and which entities, do you even extract?

The instinctive reaction is to design the perfect, complete ontology upfront. That’s exactly the trap that freezes the project.

The strategy is a not-overkill ontology. You need something flexible enough to kick off with almost no friction before you really know your domain, extending it with domain-specific detail as you explore your data.

Concretely, you use a small, fixed, generic, but extendable noun data model, known as POLE+O. Plus two core primitives, Preferences and Facts, for everything that doesn’t fit into the nouns.

You ship something that works, then add subtypes as a lightweight data-exploration step shows you where the generic types clash with your real data.

This approach lets you stand up a knowledge-graph memory layer for your own assistant without burning weeks on schema design. To build this, we first need to understand what an ontology actually is and why targeted models beat exhaustive ones.

Start Your Transition Into AI Engineering (Product)

This article showed how to design the ontology your knowledge-graph memory needs. My Agentic AI Engineering course shows the harness around it. I just released a free preview to build and run a working agent in 5 minutes.

You build a multi-agent system with two MCP servers (Research Agent + Writing Workflow), a deep research algorithm, an evaluator-optimizer loop, observability, and LLM-as-judge evals. Patterns required to ship AI.

Built for software, data engineers or scientists transitioning into AI engineering.

7 free lessons, 2 MCP agents ready for your GitHub portfolio. Part of our 35-lesson course. Rated 5/5 by 300+ students.

Start the free preview →

What Is an Ontology?

An ontology is the formal answer to 1 question. When you read the world, what do you write down as nodes, and what do you draw as edges? It specifies the kinds of things that exist in your domain, their properties, and how they relate to each other.

The ontology’s job is to map a targeted slice of the real world into the digital world. A good ontology is highly targeted to the problem you actually want to solve. If you over-model, you drown in noise and never ship. Plus, it get’s extremely expensive to extract and maintain the knoweldge graph. If you under-target, the graph doesn’t reflect the reality you care about.

Look at concrete, shipped ontologies for real-world proof. The create-context-graph domain catalog made by Neo4j publishes 22 ready-made domain ontologies. Every single one lands at exactly 10 to 12 entity types. They use a shared 5-noun base plus only 5 to 7 domain-specific nouns.

For example, the Personal Knowledge domain models the world as Note, Contact, Project, Topic, Bookmark, and JournalEntry. The Agent Memory uses Agent, Conversation, Memory, ToolCall, and Session. The lesson here is that real ontologies are small on purpose. They capture only the entities required to answer the questions the system is designed for.

So if targeted and small is the goal, why does everyone — me included — reach for big and perfect first? That’s the trap.

The Overkill Trap: Why My Knowledge Graphs Never Shipped

When I first encountered the ontology concept, I assumed I had to study my domain in depth. I thought I needed to model all of finance, for example, and design the ideal ontology before working with any real data. You can’t actually do that before you have a system running and data to look at. You just pile up assumptions that mostly turn out wrong.

I got frozen. Every knowledge-graph solution I started stayed on my laptop and never got used, because I was waiting on an ideal ontology I could never reach. Without understanding the ontology, I couldn’t even write a decent extraction step to populate it. I was deadlocked, bringing 0 value.

The breakthrough was realizing I need a couple of models that let me start generic and extend over time. As I get more data, analyze it, and actually understand my problem, the schema evolves. Let’s meet the base model that lets you start in 5 minutes instead of 5 weeks.

The POLE+O Data Model

POLE+O is a tiny, fixed, top-level vocabulary that can classify almost anything you pull out of text. It stands for Person, Object, Location, Event, and Organization [2]. It originated in law-enforcement and intelligence analysis. The Organization type was added for general-purpose entity extraction. The point of a fixed base is queryability. There are always exactly 5 base nouns to filter on, so the graph stays answerable no matter how it grows underneath.

Person covers people, aliases, and personas. Object covers physical or digital things. Location covers places, addresses, and regions. Event covers meetings, transactions, and incidents. Organization covers companies, teams, and institutions. Two or three of these catch the overwhelming majority of what a personal assistant needs.

Here are POLE+O’s five base types and the default subtypes each one ships with:

Here’s the beauty of this approach. You extend the base nouns with your own subtypes, and that’s how you tailor a generic ontology to your specific domain. It works exactly like object-oriented programming. You start from base classes you adopt without thinking. Then you subclass into specifics as your use case clarifies.

You can kick off with nothing extended and add concrete types only as you understand your data better. Neo4j’s agent-memory library uses precisely this approach. POLE+O is its default, swappable ontology.

The data-exploration workflow runs in a simple loop. First, kick off with generic POLE+O. Second, run an exploration extraction over your real data. Forget production reliability. You only care about understanding what’s there. Third, inspect the graph for clashes where the generic model lies about your data. Fourth, add or rename subtypes to fix each clash. Finally, repeat the process. You won’t get it perfect, and that’s the point. You iterate like any other AI app instead of freezing.

Look at named examples from real extraction runs. Claude Code comes back tagged as a Person when it’s clearly an Object. The “AI Engineer” conference lands as an Event when you wanted an Organization. DeepSeek is tagged a Person, not an Object.

Portugal and New York both get a flat Location label even though one’s a country and one’s a city. An agentic harness shows up as a generic Object when, for knowledge work, you’d rather have a Topic type. Each clash is a signal to add 1 subtype, not to redesign the whole schema.

POLE+O nouns and their subtypes cover the things in your world. But to fill in the gaps there are two specials tricks we have to go over.

Preferences: The Things a Noun Likes

Preferences are the second family of entities you attach to the graph. They are things a noun likes or dislikes. A Preference is a characteristic of an entity. It represents a stance. The canonical case is a person who likes, prefers, or dislikes something.

Concretely, a Preference entity looks like this:

category groups the preference, preference is the statement itself, and context optionally records when or where it applies. confidence runs from 0 to 1. The embedding makes it semantically searchable.

Make it concrete. “Loves Italian food”, “prefers dark mode”, and “dislikes long meetings” are clear examples. Each is a stable stance the assistant should remember and adapt to.

By default, a Preference hangs off the Person. That’s the most common and useful case. You can extend preferences to other objects, like an Organization’s policies, a car’s settings, or an Event’s dress code.

Because I’m building a personal assistant, I start by attaching Preferences only to the Person. This keeps the graph clean, low-noise, and small. I’ll extend it later only when a concrete use case demands it.

Preferences are the personalization layer. They act as the memory of the user’s stances. They are the “sweet sauce” that makes every future response feel tailored.

There is one issue. Plenty of useful knowledge is just an atomic fact. Forcing all of that into the ontology is how graphs explode in complexity. The fix is a deliberately generic primitive.

Facts: The Trick You Haven’t Thought Of

The Facts entity is the fallback for everything that doesn’t cleanly fit a noun or a Preference. You drop the claim into a generic Fact. This is the move that keeps the ontology small and stops you from over-thinking the schema.

A Fact is the closest thing to a classic-RAG chunk. An LLM produces each Fact during extraction. Each Fact holds a single, atomic concept which works like a charm via semantic search.

The beauty is that with facts you avoid the usual chunking errors, such as splits mid-thought, mixed concepts, and arbitrary boundaries. In reality, a Fact is a triplet. A subject, predicate, and object like “Eiffel Tower / is / 330m tall” gets embedded and stored as 1 granular unit.

Here is the shape of a Fact entity:

The triplet — subject, predicate, object — is the whole fact. valid_from and valid_until give it optional bi-temporal validity. The embedding, computed over the concatenated triplet, is what makes the fact retrievable by semantic search.

It’s confusing that we have a triplet stored as a node. But this is what it makes it flexible. We don’t worry about modeling these one-off triplets directly into the ontology, but the LLM extracts them as-is from the text.

Facts are usually wired to nothing. They have no relationships to other entities. They are retrieved only via semantic search and text search. A Fact stays in the graph but is independent of it. This works because a graph store runs vector search and graph traversal in the same query engine [4]. Which means facts are retrieved only via semantic/text search.

Facts let you ship a memory layer before you have the perfect ontology. Anything you can’t yet model degrades gracefully into a searchable atomic node instead of blocking the build. Early on, you lean on Facts. As the graph matures, claims migrate toward typed entities and edges. It costs nothing to schema and nothing to maintain when entities merge or get deleted.

What’s Next

The takeaway is the posture. An ontology is a living artifact you bootstrap from a fixed generic base and grow through a data-exploration loop, exactly like any other AI application.

If you want to see the whole strategy implemented, the fastest path is to play with Neo4j’s agent-memory SDK or its MCP server. It uses POLE+O as a swappable default, subtypes as cheap extensions, and Preferences and Facts as first-class primitives. Studying it is what made all of this finally click for me.

I’m actively migrating my own Obsidian Second Brain toward the POLE+O, Preferences, and Facts primitives. This turns thousands of files into a graph I can actually traverse, visualize, and watch evolve over time.

But here is what I’m wondering:

If you worked with Knowledge Graphs, what was your process in discovering your own ontology?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

References

1. Create Context Graph. (n.d.). Domain Catalog. create-context-graph. https://create-context-graph.dev/docs/reference/domain-catalog

2. Neo4j Labs. (n.d.). POLE+O Data Model. Neo4j Agent Memory. https://neo4j.com/labs/agent-memory/explanation/poleo-model/

3. Neo4j Labs. (n.d.). Neo4j Agent Memory. GitHub. https://github.com/neo4j-labs/agent-memory

4. Neo4j Labs. (n.d.). Why Neo4j? Graph-Native Memory Architecture. Neo4j Agent Memory. https://neo4j.com/labs/agent-memory/explanation/graph-architecture/

Images

If not otherwise stated, all images are created by the author.

I already have a second brain setup based on Obsidian, Readwise, NotebookLM, and Claude Code. I dump all my notes, research, and highlights there. Whenever I want to create content, I create a scoped wiki targeted toward the topic. I gather information from my second brain using a deep research algorithm on top of my private data and external resources via NotebookLM. The wiki is structured like the LLM Knowledge Base presented by Andrej Karpathy.

This setup fails to extract and maintain shared entities, preferences, and facts across the wiki as the knowledge base grows. For example, if the topic “Claude Code” is mentioned in 10 documents, I want to extract all the metadata about it into its dedicated folder. I want to see what other entities it relates to, such as Anthropic, San Francisco, Codex, or Gemini CLI. I also want to see how many documents mention it to rank frequency. You can do that with a pure file-based system and Obsidian, but performance degrades when your data scales past 50 documents.

The same concept applies to any unstructured knowledge base. You need a way to extract and connect knowledge from your conversations, documents, and images. This becomes essential between conversations so your agent doesn’t forget you. Instead, it provides a personalized experience. It’s also critical for context engineering to inject the right context at the right time and keep the LLM focused on relevant facts.

Most teams default to one of two memory approaches. Both collapse under real use. A file system gives you append-only logs that the agent re-reads from scratch, which fragments and rots context.

A vector index gives you fuzzy semantic recall but no merge, no identity, and no way to know if this is the same Karpathy you knew yesterday. Durable AI memory requires a structured graph to track identity and relationships [1]. Without this structure, the assistant forgets past interactions and fails to build compounding intelligence.

Knowledge-graph memory is the next step on the arc from Retrieval-Augmented Generation (RAG) to agentic RAG to agent memory [2]. Building a unified knowledge-graph memory system is hard, so most teams skip it.

During my research, I stumbled upon neo4j-labs/agent-memory. It’s a masterpiece. Who knows more about knowledge graphs (KGs) than Neo4j?

After I spent 2 days playing with it and understanding the codebase, I realized it was the perfect mental model for any agent memory system powered by KGs.

In this article, I’ll walk through the core architectural patterns of neo4j-labs/agent-memory. It features 1 graph, 3 memory tiers, the POLE+O ontology, a 3-stage extraction pipeline, a composite resolver, and the SAME_AS pattern.

By the end, you’ll have a concrete mental model. You can ship on top of their Software Development Kit (SDK) or hook it into your agent via their Model Context Protocol (MCP) server. Alternatively, you can steal the patterns and ship the same architecture on Postgres or MongoDB if a full graph database in production doesn’t make sense for your use case.

Start Your Transition Into AI Engineering (Product)

This article shows the memory layer your agent needs. My Agentic AI Engineering course shows the harness around it, and I just released a free preview that lets you build and run a working agent in 5 minutes.

You build a multi-agent system with two MCP servers (Research Agent + Writing Workflow), a deep research algorithm, an evaluator-optimizer loop, observability, and LLM-as-judge evals. The production patterns behind agents that actually ship.

Built for software, data engineers or scientists transitioning into AI engineering.

7 free lessons, 2 MCP agents ready for your GitHub portfolio. Part of the 35-lesson course. Rated 5/5 by 300+ students.

Start the free preview →

What’s Inside neo4j-labs/agent-memory

The SDK takes natural-language interactions on the write side and returns a fused memory context on the read side. Everything anchors to a single Neo4j graph. For our scoped wiki, notes and Readwise highlights about Claude Code flow in. A structured pull of what the agent knows about Claude Code, how it relates to Anthropic, and its frequency across 50 documents comes out.

At its core, there is 1 graph and 3 memory tiers joined by typed edges: short-term conversations, long-term typed entities, and reasoning traces. They’re stitched together by :MENTIONS, :TOUCHED, and :INITIATED_BY relationships [3].

The architecture contains 8 small, single-responsibility modules. The models/ module holds Pydantic schemas. The schema/ module handles Cypher migrations. The extraction/ module runs the Named Entity Recognition (NER) pipeline. The resolution/ module holds the composite resolver. The dedup/ module manages the SAME_AS pattern. The core/ module provides MemoryClient.get_context(). The mcp/ module runs the FastMCP server with 15 tools. The integrations/ module holds 9 framework adapters for tools like LangChain and LlamaIndex.

Consider an end-to-end scenario. You drop a Readwise highlight about Claude Code into your scoped wiki. The extraction/ module pulls Claude Code as an Object, Anthropic as an Organization, and Codex as an Object. The resolution/ module canonicalizes each against existing nodes. The dedup/ module checks vector similarity and either auto-merges or flags a pending :SAME_AS edge. The schema/ module commits :MENTIONS edges from the note to each entity.

Later, MemoryClient.get_context() pulls fused context across the same graph in one call. This matters concretely for the scoped-wiki agent. You can ask what you discussed last session, what you know about Claude Code, and why the agent surfaced a Codex comparison last Tuesday. The SDK answers all three against the same graph. It uses the same Cypher dialect with no cross-store join logic.

Short-Term, Long-Term, Reasoning Memory

The SDK splits memory into three layers that all live on the same Neo4j graph [3]. Short-term memory is the linear message sequence. It uses ordered :Message nodes chained by :NEXT edges, scoped to a :Conversation. Long-term memory is the typed entity graph. It uses deduplicated :Entity nodes with vector embeddings and arbitrary domain relationships.

Reasoning memory is a tree per agent run. It uses a :ReasoningTrace root with child :ReasoningStep nodes capturing thoughts and tool calls. For the scoped-wiki agent, short-term memory holds your current chat. Long-term memory holds the canonical Claude Code entity plus its relations to Anthropic, San Francisco, Codex, and Gemini CLI. Reasoning memory holds the trace of how the agent picked those specific notes to answer you.

Three relationships do the entire stitching. The :MENTIONS edge joins short-term to long-term memory. The :INITIATED_BY edge joins reasoning to short-term memory. The :TOUCHED edge joins reasoning to long-term memory. These three edges make provenance a one-hop query rather than a log-reconstruction project.

Reasoning memory is the novelty from this architecture. By storing past successful or failed thinking patterns into the memory, the agent can one-shot future similar requests or at least know not to repeat similar mistakes. Intuitively, it’s similar to Reinforcement Learning (RL), but instead of baking the optimizations into the weights, you do it at the database level.

The most important part of this architecture is the ontology.

The Ontology

The long-term memory uses a closed five-type vocabulary for its ontology known as POLE+O. It uses Person, Object, Location, Event, and Organization, borrowed from intelligence-analysis taxonomies [5]. Every entity is exactly one of these five types. Subtypes are open, but the top-level vocabulary is fixed.

In the personal assistant, Karpathy is a Person. Claude Code is an Object. Anthropic is an Organization. Your Tuesday deep-research run is an Event. San Francisco is a Location.

Type and subtype materialize as multi-tier Neo4j labels. The query builder sanitizes and PascalCases them into labels like :Entity:Person:Individual. You can search by type or subtype, making this solution highly efficient.

Using this strategy, you can extend each core type from POLE+O with your own custom domain. Other defaults are: :Entity:Location:City, :Entity:Event:Concert, :Entity:Organization:Company, etc. Here is a catalog of over 20 domains such as Data Journalism, Gaming, Personal Knowledge, and Product Management.

Entities modeled via POLE+O are nouns. The SDK adds 2 other node types beyond entities.

:Fact nodes hold every claim mentioned in the text. They’re intentionally generic so the ontology doesn’t get over-specified. They serve as a fallback when nothing else fits. You can intuitively see them as chunks of text that contain only 1 concept.

Then there are :Preference nodes that store user preferences via a SUPERSEDED_BY relationship. As agent memory is user-centric, this provides the WOW effect where the agent remembers past preferences and learns from them over time.

For the scoped wiki, “Anthropic developed Claude Code” is an edge. “Claude Code 1.0 shipped in 2025” is a :Fact. “I prefer agent-harness comparisons over pure benchmarks” is a :Preference.

Extraction: From Raw Text to Typed Entities

The SDK runs entity extraction as a speed-versus-accuracy ladder. It uses spaCy for fast statistical NER. It uses GLiNER and GLiREL for zero-shot extraction. It uses an LLM stage for cases that need real semantics and to extract the relationships between them [6].

Each stage maps its outputs back to POLE+O types. It uses explicit merge strategies when 2 extractors disagree. When you drop a Readwise highlight about Claude Code into your scoped wiki, spaCy lifts proper nouns like Anthropic and San Francisco. GLiNER catches domain entities like Claude Code and Gemini CLI. The LLM stage only fires when the previous 2 stages leave ambiguity, or when the model needs to extract relationships.

Routing every mention through an LLM would multiply extraction cost massively for marginal recall on rare entities. The ladder pushes high-confidence cases to cheap models. It escalates only ambiguous mentions to the zero-shot models and reserves the LLM stage for when real semantics matter.

The real problem is at the normalization step.

When Two Mentions Are the Same Entity (And When They Aren’t)

Resolution and deduplication are 2 different problems. Resolution sets a canonical string property on an existing reference. Deduplication decides whether a new node gets created at all. Conflating them is how graphs end up with 3 Anthropic nodes that none of your queries find together [7].

Resolution runs 3 strategies on the name field in cost order. Exact matches existing canonical strings. Fuzzy uses RapidFuzz string similarity for surface variants like “A. Karpathy” and “Karpathy, Andrej”. Semantic falls back to embedding similarity for cases like “the founder of Eureka Labs”. It only matches between nodes of the same type, meaning a Person only resolves against Person candidates.

After resolution runs, two mentions like “Apple” and “Apple Inc.” end up with different surface names but the same canonical name. That’s why a second step is needed. Deduplication looks at the semantics, not just the name.

For deduplication, the SDK uses vector and fuzzy similarity across the entire node content. This ensures the node is actually the same, not just a name coincidence. In other words, this avoids false positives. Using vector and fuzzy search, the SDK computes a score.

Scores at or above 0.95 trigger an auto-merge. Scores below 0.85 create a new node. Scores between 0.85 and 0.95 don’t silently merge. Instead, they create a :SAME_AS edge with a pending status. This flags the edge for a human or downstream agent to resolve later. This pattern stops “Jensen Huang the NVIDIA CEO” from merging with “Jensen Huang the Taipei dermatologist” just because their embeddings landed 0.91 apart [7].

A false merge is silent and unrecoverable. A false split is noisy but recoverable. You can’t undo a false merge without re-ingesting from the raw source data. That’s why you should leave uncertainty to a human.

Zooming into the Retrieval Algorithm

Because all three tiers live on one graph, a single retrieval can compose vector similarity over :Entity embeddings, multi-hop expansion over typed relationships, time-ordered :NEXT conversation walks, and reasoning-trace lookups via :INITIATED_BY and :TOUCHED joins. All of these run as steps in the same Cypher query. Neo4j 5.20 introduces db.index.vector.queryNodes, making vector similarity a first-class graph operation [4].

When you ask what you know about Claude Code, how it relates to Codex and Gemini CLI, and why you looked at it last week, the agent fuses three things in one pull. It uses vector similarity over your Readwise highlights to surface relevant passages. It uses a multi-hop traversal of :DEVELOPED_BY and :COMPETES_WITH edges to bring in Anthropic and Codex neighbors. Finally, it uses an :INITIATED_BY jump back to the prior conversation that discussed agent harnesses. There’s no cross-store join logic and no orchestrator.

From our tests, the library leaves the context construction to the user of the SDK. In other words, you get the whole output from the graph, and it’s your responsibility to further compress it before passing it to the LLM.

What’s Next

The neo4j-labs/agent-memory architecture is more complex than what this article covers, but this is the core idea behind it. I’ll cover other components in more depth in future articles, including designing the ontology and keeping your knowledge graph clean over time.

I think this open-source repository is a perfect blueprint you can take to build your own agent memory solution, even with Postgres or MongoDB, to avoid keeping multiple databases in production. Still, Neo4j is probably the best choice for data mining and exploration.

For small to medium-scale projects with thousands of nodes and short hop traversals, I’d probably build my own agent memory solution from scratch on top of Postgres or MongoDB. I’d reach for Neo4j as an internal tool within my organization, or when the scale or complexity becomes too large for Postgres or MongoDB.

But here is what I’m wondering:

How are you handling agent memory today? Flat files, a vector index, a knowledge graph, or something stranger?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

References

1. Seale, T. (n.d.). This week Anthropic dropped Claude Sonnet 4.5. LinkedIn. https://www.linkedin.com/posts/tonyseale_this-week-anthropic-dropped-claude-sonnet-activity-7379787334398926848-iVOE/

2. Monigatti, L. (n.d.). The Evolution From RAG to Agentic RAG to Agent Memory. Leonie Monigatti. https://www.leoniemonigatti.com/blog/from-rag-to-agent-memory.html

3. Neo4j Labs. (n.d.). Understanding the Three Memory Types. Neo4j Agent Memory. https://neo4j.com/labs/agent-memory/explanation/memory-types/

4. Neo4j Labs. (n.d.). Why Neo4j? Graph-Native Memory Architecture. Neo4j Agent Memory. https://neo4j.com/labs/agent-memory/explanation/graph-architecture/

5. Neo4j Labs. (n.d.). POLE+O Data Model. Neo4j Agent Memory. https://neo4j.com/labs/agent-memory/explanation/poleo-model/

6. Neo4j Labs. (n.d.). How Entity Extraction Works. Neo4j Agent Memory. https://neo4j.com/labs/agent-memory/explanation/extraction-pipeline/

7. Neo4j Labs. (n.d.). Entity Resolution and Deduplication. Neo4j Agent Memory. https://neo4j.com/labs/agent-memory/explanation/resolution-deduplication/

Images

If not otherwise stated, all images are created by the author.

I needed a TypeScript harness for my latest book code. It required a Terminal User Interface (TUI), an agent loop, tools, Model Context Protocol (MCP) support, skills, and slash commands. I will be honest with you. I first tried to vibe code this project.

As I knew what I was looking for, it worked. Until it didn’t. The code was working until you started looking more closely at the details. Only the first 20 characters were rendering inside the TUI, and the skills weren’t invoked by the agent loop.

So I deleted the whole code base and started over with a new strategy.

The cost of vibe coding isn’t abstract. It’s the next feature you can’t ship because you’re debugging a slash-command renderer that looked finished. This is what most people get wrong. Output that compiles and looks done breaks the moment you reach for the rough edges.

I divided the harness into tasks. I one-shotted the barebones version, which was just a TUI plus an agent loop with bash, grep, and a todo tool. Then I layered MCP, skills, and slash commands as separate features.

You can’t one-shot whole applications. You can one-shot big features if you scope them right and run them through a real engineering process.

This is known as agentic coding. Not vibe coding. You’re using agents to write the whole codebase, but you are still the mastermind behind everything.

But I wanted more. I wanted to automate this process. But with a single constraint in mind: “the code should HAS to be good”.

That’s why I built Squid. It’s an opinionated six-agent Claude Code setup available at iusztinpaul/squid. It ships features the way a real software team ships them.

Squid has already shipped our content-automation tool, expanding it from articles to posts, notes, threads, and messages. It shipped the book’s code data pipelines and TypeScript harness.

In this article I will show you how it works.

The concrete blueprint relies on a specialized team and an e2e lifecycle.

Start Your Transition Into AI Engineering (Product)

Squid applies the multi-agent pattern to coding. My Agentic AI Engineering course applies it to writing, and I just released a free hands-on lesson that distills the whole system.

You build a multi-agent system composed of two FastMCP servers (Deep Research + LinkedIn Writer) orchestrated by a harness, plus an observability and evals layer on top. The shift from classic backend/frontend stacks to MCP servers and harnesses is the pattern shaping modern agentic AI.

Built for software and data engineers moving into agentic AI engineering.

Part of the 35-lesson course. Rated 5/5 by 300+ students. First 7 lessons free.

Start the free lesson →

The Six Agents Engineering Team

The system contains six agents. No agent both writes code and decides whether the code is correct.

The product manager agent manages the tasks and ensures the feature adheres to the software architect’s specifications. It takes a raw feature specification, writes or updates an Architecture Decision Record (ADR) for non-obvious choices, and splits the feature into ordered tasks. It also maintains the Domain-Driven Design (DDD) glossary so vocabulary stays consistent between the business and engineering.

Note how, because Claude can easily handle both PM and software architecture work, we decided to merge these roles together. We did this to avoid fragmenting the context just to follow a standard human process. Ultimately, planning should be closely aligned with the software architect’s vision. In human processes, dividing these two responsibilities often created more issues than solutions.

The software engineer agent uses red-green Test-Driven Development (TDD). It writes the failing test, writes the minimal code to pass it, and then refactors. The software engineer uses direct command-line interfaces (CLIs) like git, mongosh, and gh. It never uses MCP wrappers. CLIs are more flexible because they tap directly into the power of bash. Plus, LLMs have seen considerably more bash code than MCP wrappers during training.

The tester agent specializes in the adversarial end-to-end edge-case pass. It catches false-confidence claims where the software engineer says the tests pass. It does this by reading every acceptance criterion against concrete evidence, like the test name, file lines, and command output.

The pull request reviewer agent performs a diff-only review. It looks for dead code, duplication, missing test coverage, and documentation adherence. It does a narrow performance review on hot paths only. It’s explicitly told not to micro-optimize one-off scripts.

The on-call agent loops on the Continuous Integration (CI) pipeline until it passes. In an earlier iteration, the CI check lived inside the software engineer and tester loop, and it got skipped constantly. Promoting it to a dedicated agent invoked by the orchestrator increased the probability the step runs.

The self-improve agent is an optional meta agent. After the feature is done, while looking over the results, the human can run the self-improve agent to scan the run for high-signal lessons and propose updates to the agentic coding layer that consists of CLAUDE.md, skills and subagents. This is a double-edged sword. It can constantly improve your workflow or quickly degrade it if you are not careful. That’s why it’s incredibly important that this step is gated by a human.

The secret sauce is in anchoring the agents into your own documentation.

Keeping Up With Documentation: ADRs & DDD Glossary

The ADR directory acts as compressed architectural memory across runs. Every non-obvious choice regarding the datastore, synchronization defaults, authentication boundaries, or dependency lock-in ships with an ADR. These records include the status, context, decision, and consequences at docs/adr/<NNNN_title>.md. The product manager reads the directory before grooming a new feature, so decisions stay consistent across feature branches.

The DDD glossary gives shared vocabulary between the business and engineering at docs/glossary.md. It enforces one canonical name per concept. Code identifiers, OpenAPI schemas, database columns, and customer-facing interfaces all use the term exactly as it appears there. This gives Claude Code business context, not just code context, properly anchoring your code in your domain. The software engineer, tester, and pull request reviewer all reason about the same domain.

I have an honest caveat. The agents still under-use both the ADRs and the glossary. The spine exists, but I am still working on getting the agents to lean on it consistently.

Now the agents have the context they need to execute a feature from a raw specification all the way to a merged pull request.

The Night Skill. The End-To-End Workflow.

The /night skill takes one input, which is a feature specification written by the human, and produces one output, which is a merged pull request with green CI. Everything in this section sits between those two endpoints.

The /night pipeline is a long-running lifecycle. That’s why it’s called the “night” skill. It’s scoped to run for hours at a time, often with multiple pipelines in parallel.

It has two human checkpoints and five retry caps, while everything else is automated. The orchestrator acts as a manager. It never writes code itself, never runs tests itself, and never reviews the diff itself. It launches agents and enforces human validation.

After a human carefully writes a detailed feature specification, it calls the /night skill, which creates a new branch and worktree. The product manager reads the glossary and ADR directory, updates or writes a new ADR if needed, and splits the feature into a task plan.

Then we hit the first human gate. The user approves the plan, optionally sharpened by the /grill-me skill. The /grill-me skill is inspired by Matt Pocock’s work, which forces the agent to ask sharp questions back about anything fuzzy in the plan, such as interfaces, modularization, or new tools. This conversation is the line between vibe coding and agentic coding.

Next is the inner loop per task. The software engineer implements the code, the tester verifies it, and failures route back to the software engineer. This loop is capped at 5 attempts. Convergence is mostly mechanical through a run, fail, fix, and run cycle.

The product manager then performs an acceptance review on the whole feature from the user’s perspective. Rejections are packed into a single task back into the inner loop. This is capped at 3 attempts, because judgment-call loops are where Claude Code spirals.

Next, we repeat a similar loop using the PR reviewer agent, which looks at the diff, with a maximum of 3 attempts to avoid perfectionism. Adding a maximum number of attempts here is critical, because during review an LLM almost always has something else to say.

After the push, the on-call agent watches CI with a maximum of 5 attempts, routing failures back to the software engineer.

When the CI is green, we notify the user (e.g., via Slack) that the PR is ready for review. Optionally, based on any potential issues found while running the /night skill, we run self-improve to propagate that into your memory.

The /night lifecycle. Two human gates, five retry caps, everything else automated.

Beautiful! With this process I one-shot most of the features I am working on. And when it’s not a one-shot, I’m typically 95–99% there by the time I review the PR.

How the Tester Stopped Re-Running What the SWE Already Ran

The biggest problem with the e2e workflow above is that it’s slow and redundant. I preferred that over generating AI slop that I have to manually review and fix.

Still, there are a few tweaks that we can make to the workflow to improve speed and efficiency.

For example, when the tester re-ran the linter, type checker, formatter, and the happy-path suite that the software engineer had already run, we paid for everything twice. This was the number-one source of having a system that works but is too slow to use.

To fix this, the tester now accepts the software engineer’s reports for formatting and happy-path tests. It only runs the adversarial end-to-end edge-case pass itself. This covers the part the software engineer can’t credibly self-verify. Trust is bounded. Intuitively, I realized I’d started shifting the Tester toward QA-style practices, rather than just running simple tests.

I am still iterating on optimizations. For example, I want to route some subagents to Claude Sonnet models instead of Claude Opus. I also plan to narrow toolsets per role to reduce reasoning failures.

Also, depending on what you are working on, you might want to use the system more as a fast, snappy assistant than as a long-running workflow that prioritizes correctness above all.

Day vs. Night: Two Orchestrators, One Team

That’s why we have two pipelines running the same agents. The /night skill is the full lifecycle. It’s long-running, set-and-forget, has two human gates, and runs while you are away from the keyboard or working in parallel.

The /day skill is the lean inner loop. It runs the software engineer, the tester, and human commits for surgical edits. It skips product manager grooming, the pull request reviewer, and the on-call agent.

There is a concrete use case for the /day skill. When I read a merged pull request and find code I don’t like, the /day skill runs the stripped software engineer and tester loop to apply targeted edits. Then the on-call agent cleans up any CI fallout. This is the surgery that keeps the system from becoming a black box.

Both pipelines have one thing in common. The human is in the loop on purpose, not as a fallback.

Why Code Templates Are a Waste of Time in 2026

Most teams are still scaffolding from cookiecutter templates that were outdated the day they were committed. This is a maintenance tax disguised as productivity. Squid stops paying that tax. Technology moves fast enough that any frozen template’s frameworks, tooling, interfaces, and opinions all need their own maintenance pipeline. That’s only worth it if one template fans out across dozens of projects.

A Copier or cookiecutter template isn’t free. I tried scaling one across Python, TypeScript, and Go. I watched the project balloon into a maintenance burden where most files would never be used. Maintaining a template engine to support multiple stacks is a full-time job.

Asking Claude Code to copy from the last project fails too. It propagates the technical debt baked into the source codebase. You inherit the mess, not the ideal state.

The real shift relies on markdown, not Jinja. I call these agentic templates.

You encode good practices as skills and CLAUDE.md files. Fundamentals like clean architecture, CI/CD discipline, testing patterns, and development cycles rarely change. When they do change, you edit prose instead of regenerating from a template engine that quickly slides into dependency hell.

Tooling stays dynamic. You don’t pin framework versions inside a template. You keep a decision tree of allowed choices and let the agent pull the latest interfaces on demand via Context7 at scaffold time.

Project structure can’t be templatized. The anti-pattern organizes by type, putting files into agents/, nodes/, schemas/, and tools/ directories. One business module’s logic ends up scattered across four folders, forcing both humans and the agent’s context window to thrash.

The correct pattern organizes by actionability, keeping one bounded context per directory. Each domain owns its own types, store, Application Programming Interface (API), and prompts. That’s locally readable, easier to maintain, and easier for the agent to reason about.

Because we describe the structure in Markdown files instead of cookiecutter templates, we can define it like this:

Avoid global dumping grounds like utils/ or helpers/. Avoid a root-level types.py grab bag. Avoid grouping tests by type.

The /scaffold skill acts as an interactive bootstrap. An AskUserQuestion prompt drives a tight decision tree covering project identity, layout, components, backend, frontend framework, infrastructure, agent team, tracker, ADR and glossary opt-ins, and external services. A deterministic table picks only the matching specifications from the specification library. Unused categories never enter the context. The skill writes a tailored CLAUDE.md brief, lays down an empty folder skeleton, and hands off.

Then, based on the agentically generated template, you can use /night or /day to start writing real code.

Open-Sourcing Squid

I don’t want to keep Squid for myself. I want to share it with the community to learn from and contribute to.

Thus, I am open-sourcing Squid.

You can install it as a Claude Code plugin:

/plugin marketplace add iusztinpaul/squid
/plugin install squid@squid

I want you to try it, build something awesome with it, and if you like it, contribute back:

Check the full codebase

Still, here is what I’m wondering:

What is your agentic coding setup? How is Squid different from your own approach?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

Images

If not otherwise stated, all images are created by the author.

I gave this talk twice in one month: at O’Reilly’s Context Engineering Event and at Abi Aryan’s Maven course on LLM inference at scale. After being blasted with questions, I realized something: GraphRAG isn’t a retrieval algorithm, it’s a data modeling problem.

Powering agents with knowledge graphs (KGs) and ontologies is still an unsolved problem. All the engineers I spoke to want GraphRAG, but don’t know how to implement it.

But at its core, we should ask a different question. Why do we even need GraphRAG in the first place? Why complicate our solution over a simple RAG system?

There are three core reasons.

First, you face context rot. As the context window fills, the signal-to-noise ratio collapses. The LLM degrades.

You pay for this degradation in quality, cost, and latency [1].

Second, you face data fragmentation. In the agent era, your data lives in silos most builders share: documents, notes, research, emails, and text messages. We are no longer lucky enough to have all the data nicely stored in a single database.

Third, the agent’s unified memory naturally maps to a knowledge graph (KG). People have preferences and experiences. They went into specific locations, met with other people, or have a list of items to do. Things get trickier when “Arthur told Felix that his favorite coffee shop is in the center of Timisoara”, but after two months “it moved to Lisbon”. You need to start tracking relationships between people, locations, and most especially how these relate in time.

GraphRAG solves all three.

This is a data modeling problem, not a retrieval algorithm. It took a painful LangChain detour and a hard MongoDB RAM conversation to settle that for me. You need an ontology.

By the end of this article, you will learn about ontology-first design, the three extraction modes, append-only data models, and hybrid retrieval joined by Reciprocal Rank Fusion (RRF). Finally, you will see how to expose the GraphRAG engine as a unified memory layer via an MCP server to power your agents. In other words, how to do agentic GraphRAG.

Before walking through the architecture, let’s understand why the story has to start from the ontology.

Build Your Own Multi-Agent System Free Workshop (Product)

This article shows what an MCP-served unified memory looks like end to end. If you want to actually build agentic systems with MCP servers like this, I open-sourced a hands-on workshop for that.

Two MCP servers from scratch: a Deep Research Agent (Gemini + Google Search grounding) and a Writing Workflow with an evaluator-optimizer loop.

Packaged with slides, a ~2-hour video, runnable reference code, and an “implement-it-yourself” skeleton via agentic coding best practices (25 tickets, one orchestrator skill, and two agents: SWE and tester).

Originally presented at the AI Engineering Conference Europe. 200+ stars on GitHub. Free.

Go to workshop →

Why the Story Starts From the Ontology

Whenever you need to connect dots across a corpus of multiple documents rather than find the most relevant paragraph, you go for GraphRAG. Knowledge is stored as entities and edges.

You traverse connections rather than find similar text.

An ontology is a collection of classes and the relationships allowed between them. If you come from object-oriented programming, you already have the right intuition.

Throughout this article, we will build a digital twin. My favorite example. We will define a Global Ontology of six entity types organized into two sub-ontologies.

The data pipeline deterministically constructs the Document Ontology. It contains DOCUMENT and CHUNK nodes. It uses PART_OF, NEXT, REFERENCED, and MENTIONS edges.

The LLM extracts the Person Ontology. It contains PERSON, TASK, EPISODE, and PREFERENCE nodes. It uses RELATED_TO, TODO, EXPERIENCED, and HAS edges.

The schema is flexible. You define it for your business case. Every section after this one assumes these exact node and edge labels.

Skipping the ontology carries a heavy cost. I tried LangChain’s MongoDBGraphStore, which lets the LLM extract entity and relationship types freely. Five documents produced 17 node types and 34 relationship types.

This included part_of, Part Of, and part of as three separate types. The underlying data model does not enforce a schema at the storage layer.

With an ontology, the LLM can only extract what you defined. The constrained scope also allows you to use cheaper extractor models.

That’s why GraphRAG is the right tool when you have a clearly defined schema. It works when you need to identify relationships.

It reduces hallucination on complex queries that span interconnected facts. Domains where knowledge graphs naturally fit are legal, medical, financial, business operations, productivity tools and in my opinion, the crown jewel: personal assistants. With a KG, you can naturally build the unified memory of your personal assistant to properly remember what you like, what you did, and what you have to do, all anchored in time.

For example, Palantir built its empire using ontologies. Google uses KG to power its search, and Microsoft uses it in its internal ops tools.

With the ontology defined, the next architectural choice is the shape of the graph itself and how to extract those entities from raw text.

RDF vs. Property Graphs, and the Three Extraction Modes

Every graph is structured as a collection of (entity, relationship, entity) triplets. But there are two ways to attach data to each entity or relationship instance, known as Resource Description Framework (RDF) and labeled property graphs.

RDF attaches each piece of metadata as another triplet. The graph explodes in size. Property graphs attach metadata as JSON on the entity or relationship.

In practice, GraphRAG and agents use property graphs [3].

Now, during extraction, where we actually map data into our (entity, relationship, entity) triplets, plus their corresponding data, we have three core methods.

Structured extraction is schema-guided. The LLM outputs entities per the Person Ontology.

Semi-structured extraction uses metadata and lineage without an LLM. You parse the email’s links and attachments.

Unstructured extraction uses an LLM without a schema. The LLM invents its own labels. This is useful for discovery, not for grounded retrieval. In other words, we use the LLM to extract triplets without an ontology. Exactly what we said to avoid in the previous section.

Here is the data-source mapping for the Person Ontology of the digital twin:

The Document Ontology can be completely done through semi-structured mechanics, since we already know what document each chunk comes from, the author of each document, and the references between them.

💡 A student asked about open-domain extraction. Exploratory extraction is great early on when you are figuring out what ontology makes sense for your data. You can use zero-shot Named Entity Recognition (NER) models like GLiNER for that exploratory phase [4]. Which you can easily run locally without having powerful inference hardware. Without that discipline, the output becomes unusable noise within tens of documents. A constrained scope lets you swap the frontier model for a small fine-tuned extractor like Gemini Flash Lite, Claude Haiku or even better, use Liquid open-source models fine-tuned on your ontology.

These extraction modes feed directly into a five-component system that turns raw documents into queryable memory.

The Five-Component Architecture

The input consists of heterogeneous documents scattered across multiple silos. The output is a single queryable knowledge graph. The agent can search and write back to it via two tools.

Everything in between is plumbing built to serve that one job.

The data pipeline gathers from URIs, notes, emails and Google Drive. It normalizes everything into a document collection written to a warehouse.

The memory pipeline turns documents into knowledge-graph objects and writes them into the unified memory modeled as a KG.

The KG is the queryable artifact. The agent communicates with the knowledge graph via an MCP server that exposes search and write tools. If you are building in Python, choose FastMCP over the native MCP SDK, as it’s much easier to use and offers a better developer experience.

The search_memory family of tools brings only the slice the agent needs into the context window. The write_memory tools run the same data + memory pipelines on demand on a conversation or URI instead of running them in batch mode [5].

Ultimately, we connect the MCP server to a harness such as Claude Code or Codex, where we inject custom business logic on how the tools should be used through a family of assistant-memory and assistant-learn skills.

For 2-3 hop traversals, Postgres or MongoDB handle documents, vectors, and graph-lookup in a single piece of infrastructure [7].

Reach for Neo4j only when deep traversals or specialized graph algorithms are core to the product [8]. Or a good trade-off is to use it internally just for data exploration. Do not design for Google scale when you are processing thousands of documents.

The memory pipeline sits at the core of this architecture, transforming raw documents into the exact triplets the rest of the system queries.

The Memory Pipeline

The memory pipeline cleans the incoming document.

Next is optional chunking. If you can avoid chunking, avoid it. It introduces problems and is more about RAG-era reflexes than a necessity. You always have to customize the solution based on your data and try to introduce as little complexity as possible.

Next, the graph extractor emits triplets. You should use Pydantic-style schema descriptors so the LLM knows how each field should look.

Normalization is the most important step. You track the evolution of a single entity over time. Do not allow multiple versions of the same person to exist. The system re-uses the same canonical ID across extractions. New metadata and new relationships layer on top [9].

Finally, you embed the relevant fields for semantic search.

Now, let’s look at the core ways of data models you can use to store your KG.

Single Mutable Collection vs. Append-Only Log Data Models

There are two main approaches on how you can model your collections: as an append-only log or as a single mutable collection. Both have their pros and cons.

The append-only log consists of two collections: an append-only log and a queryable materialized view.

The system appends every event to an immutable log. A periodic materialization step squashes all events for the same ID into one canonical record.

You get versioning, temporality, and reversibility for free. You pay in RAM and operational complexity. As RAM is the most scarce and costly piece of hardware for hosting databases, this quickly translates into larger compute costs.

The single mutable collection approach drops the log. Each extraction directly upserts into the queryable collection.

You get simpler ops and real-time visibility, but the temporal audit trail is gone. Pick the single collection if operational simplicity and reduced costs beat time-travel.

Pick the two-collection append-only approach if you genuinely need an audit trail. Append-only collections never delete and never update. The same ID can appear multiple times across extractions, reflecting updates of an entity or relationship instance across the KG.

You can replay history up to a point in time, soft-delete, and revert a bad extraction. Materialization squashes all logs sharing an ID into one canonical entity.

An intuitive way of comparing the two methods is that the single mutable collection option is the same as the materialized view of the append-only option. Thus, one option comes with an append-only log, which comes with versioning and temporality, while the other doesn’t.

How Would This Look Within the Digital Twin?

Each log event lands with an auto-generated ObjectId plus a single chunk_id and source_document_id pinning it to one origin, with no embedding because nothing has been merged yet into the final instance. Materialization groups events by (name, type) for nodes and by the (source, kind, target) triplet for edges, swapping the ObjectId for a deterministic composite ID that is the merge key, unioning every contributing document into a sources array, and embedding each canonical entity once.

Nodes and edges share a single collection, separated only by a kind discriminator. So within our MongoDB implementation, $graphLookup walks source_node_id → target_node_id recursively without joining across collections.

A student asked about community detection and isolated nodes. Once materialization runs, the system computes communities over the canonical node collection. An isolated node is just a singleton community. Filter or keep it based on your use case. Postgres and MongoDB handle hundreds of millions of small records. They can also scale vertically easily through sharding by partitioning on the entity and relationship IDs.

Now, let’s finally understand how we can query the KG and plug it into an agent.

Finally...Let’s Understand the Retrieval Algorithm

During retrieval, we use a hybrid index.

Text search uses exact keywords. Semantic search is meaning-based. Graph search is a multi-hop traversal across the typed edges.

Communities are an optional fourth index for topical clusters.

GraphRAG retrieval is a two-stage move [10].

Stage 1 runs text and semantic search. It merges results with Reciprocal Rank Fusion (RRF). Apply a cutoff to get your entry points [11].

Stage 2 walks 2-3 hops across the typed edges to expand the result set.

During retrieval, GraphRAG’s addition over RAG is this multi-hop step, after the RRF merge, which is standard for most RAG systems.

Still, there are two important details to highlight. There’s bottom-up, which expands entities for depth, while top-down hops across communities for a high-level overview [2]. This translates to a trade-off between context size, latency and performance.

Now, to close the loop, let’s connect everything to an agent.

The Cherry on Top: Agentic GraphRAG

GraphRAG becomes agentic when an agent gets to write to and search the knowledge graph autonomously [5].

The agent dynamically writes queries against the materialized knowledge graph using a family of search_memory tools. The write_memory family of tools runs the data and memory pipelines on the current conversation or any other type of document. These tools are exposed to the agent via the MCP server, implemented in FastMCP.

This differs from the five-component architecture explained earlier: this time, the agent decides when to search/write to memory.

The search tools can directly implement the text + semantic + graph-search algorithm programmatically, or let the agent write the query code on-demand, which gives more flexibility at the cost of potentially less optimal code.

As for the write tools, allowing the agent to ingest the current conversation ensures continual learning by dynamically tracking the user’s preferences, to-dos, experiences and more.

At the moment, harnesses such as Claude Code use the filesystem to implement the memory layer. But as the data grows, gets more complex, or we have to operate under strict cost/latency requirements, we will need more powerful solutions than just hoping the agent will figure it out through progressive disclosure.

What’s Next

In this piece, I presented only the high-level architecture and strategies around GraphRAG.

The issue is that when you start diving into each component, such as normalization, extraction, embedding or data modeling, you will realize that everything is extremely custom to your own data and use case.

This is especially true because GraphRAG is still in its early days, where there is no clear plan of attack.

That’s why I am actively working on a new book on how to implement a personal assistant from scratch (yes, together with Maxime Labonne!), where we will explore building a memory layer stage by stage: RAG, then GraphRAG, with an AI Evals layer on top to measure the actual gain in performance when introducing GraphRAG. As soon as I have more details on this, I will let you know.

But here is what I’m wondering:

Are you using a single database (Postgres / MongoDB) or splitting graph and vector workloads across specialized systems (Neo4j + Pinecone)?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

References

1. Anthropic. (n.d.). Effective Context Engineering for AI Agents. Anthropic. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

2. Larson, J. (2024, April 2). GraphRAG: Unlocking LLM Discovery on Narrative Private Data. Microsoft Research. https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/

3. Negro, A., Kus, V., Futia, G., & Montagna, F. (n.d.). Knowledge Graphs and LLMs in Action. Manning. https://www.manning.com/books/knowledge-graphs-and-llms-in-action

4. Neo4j Graph Data Platform. (n.d.). How Entity Extraction Works. Neo4j Agent Memory. https://neo4j.com/labs/agent-memory/explanation/extraction-pipeline/

5. Monigatti, L. (n.d.). The Evolution From RAG to Agentic RAG to Agent Memory. Leonie Monigatti. https://www.leoniemonigatti.com/blog/from-rag-to-agent-memory.html

6. Govindarajan, V. (n.d.). OpenClaw Architecture - Part 3: Memory and State Ownership. The Agent Stack. https://theagentstack.substack.com/p/openclaw-architecture-part-3-memory

7. Iusztin, P., & Rodrigues, J. (n.d.). How We Killed Our RAG Pipeline.

8. Neo4j Graph Data Platform. (n.d.). Why Neo4j? Graph-Native Memory Architecture. Neo4j Agent Memory. https://neo4j.com/labs/agent-memory/explanation/graph-architecture/

9. Neo4j Graph Data Platform. (n.d.). Entity Resolution and Deduplication. Neo4j Agent Memory. https://neo4j.com/labs/agent-memory/explanation/resolution-deduplication/

10. Hedden, S. (n.d.). How to Build a Graph RAG App. Towards Data Science. https://towardsdatascience.com/how-to-build-a-graph-rag-app-b323fc33ba06/

11. Arancio, J. (n.d.). Comment on Hybrid RRF Retrieval Pipeline. Substack. https://substack.com/@jeremyarancio/note/c-205294494

12. Liu, J. (2025, May 19). There Are Only 6 RAG Evals. jxnl. https://jxnl.co/writing/2025/05/19/there-are-only-6-rag-evals/

13. Zhang, B. (2026, January 22). Scaling PostgreSQL to Power 800 Million ChatGPT Users. OpenAI. https://openai.com/index/scaling-postgresql/

14. Govindarajan, V. (n.d.). OpenClaw Architecture - Part 2: Concurrency, Isolation, and the Invariants That Keep Agents Sane. The Agent Stack. https://theagentstack.substack.com/p/openclaw-architecture-part-2-concurrency

Images

If not otherwise stated, all images are created by the author.

Most AI demos work. Most AI products don’t. This series is a collection of interviews with engineers who shipped AI agents to production, covering the stacks they chose, the architectures they regretted, and what actually held up at 3 am.

This is an interview with Michael Maximilien, former CTO and Distinguished Engineer at IBM and Chairperson of the Board of the NodeJS Foundation. Now, the founder and CEO of ClawMax.ai, an AI agent orchestration platform powered by OpenClaw and the creator of weave-cli, an open-source tool for shipping Retrieval-Augmented Generation (RAG) systems.

Watch our full interview on YouTube ↓

Michael Maximilien spent a year building RAG systems for customer after customer. Every new project required navigating dozens of moving parts. He had to pick a vector database, select an embedding model, chunk the data, ingest it, search it, and iterate.

“I was doing this a lot and I wasn’t getting the results I wanted.” — Max

The failures were concrete. Halfway through an ingestion run, Milvus would run out of memory. Two collections made it in. The third was broken. Without a checkpoint or resume function, he had to recompute everything from scratch.

“The experiment doesn’t just run, it fails. You have to be able to pick up from the failure.” — Max

Another failure mode involved manually comparing Weaviate against Milvus. One configuration typo could lead to drawing the wrong conclusion.

“You might end up thinking Weaviate is better than Milvus when actually your comparison was wrong.” — Max

This manual flywheel stole time from actually helping his customers ship their products. He burned days on reset and re-ingest cycles that failed halfway. Worse, he produced results he could not trust.

Most teams treat RAG as a simple setup task. They picked a vector database because it trended online. They pick an embedding model because OpenAI is the safe default.

They guess a chunking strategy, guess the top-K retrieval parameters, and ship it. Then they spend the next six months vibe-checking the system.

Users complain. The team swaps a configuration knob. Nobody knows if it actually helped because nothing was measured.

“There’s a lot of steps.” — Max

You lose the working system you thought you had. You burn weeks debugging silent ingestion failures because no trace exists.

Customer trust evaporates when the same question gets three different answers across releases.

Max took the opposite bet. He built Weave CLI: a unified command-line tool for RAG over eleven vector databases.

It features first-class observability implemented with Opik, an open-source evaluation and optimization tool, baked in from the first commit. You can try out their managed platform for free here for 25k spans/month.

By the end of this case study, you will understand how to unify your RAG stack so that switching a database, an embedding model, or an agent is merely a config change. You will learn how to measure everything, so every switch is tracked, evaluated and compared. Ultimately, you will learn how to benchmark your solution against multiple parameters to find the best configuration for your problem.

“There’s no one solution. You iterate and evaluate.” — Max

But first, let’s understand what Weave CLI is and how it works.

Understanding the System Architecture of Weave CLI

Weave CLI wraps 11 vector databases behind a single interface. From the outside, it looks and feels like any other RAG system. On the ingestion side, it populates the chosen vector database with chunks, metadata, and embeddings. On the query side, it takes natural-language questions and returns top-k ranked chunks that an agent can use to create an answer with citations.

What makes Weave CLI special is that everything is swappable via a configuration file: the vector database, the embedding model, the chunking strategy, the query agent, the RAG agent that interprets the chunks and so on. With the goal of making it very easy for you to benchmark, iterate on and improve your RAG solution.

Weave CLI is composed of seven core components, each swappable by configuration.

The user-facing component is the Cobra-based CLI and the interactive REPL. Weave stack sits underneath as the deployment layer. It brings the whole system, the databases, up or down with a local Docker/Podman Compose fallback.

Behind that surface sits the intelligence layer. Ten built-in agents share an AgentChain sequencer. Agents are used both within the CLI and during ingestion. Weave CLI supports RAG, QA and summarization agents, but what’s more interesting is during ingestion. For example, you describe your data, the SchemaAgent proposes a collection schema and a vector-database fit, the ChunkingAgent recommends a chunking strategy and an embedding provider is picked to match. The Executor drives a seven-step orchestration covering query analysis, planning, user confirmation, execution, reporting, display, and evaluation metrics.

The data layer is built around the VectorDBClient interface in src/pkg/vectordb/interfaces.go. It is cleanly split into four sub-interfaces: CollectionOperations, DocumentOperations, QueryOperations, and SchemaOperations. A package-level factory registry in factory.go registers all 11 adapter sub-packages using the ports-and-adapters pattern.

Still... There is a trade-off to this design. A unified interface is a lowest-common-denominator by construction, so if you need PGVector’s transactional semantics or Neo4j’s native graph traversal as first-class features, a unified adapter costs you that expressiveness.

On top of the vector-database layer sit five embedding providers: OpenAI, sentence-transformers, Ollama, Cohere, and Voyage. The ingestion pipeline runs alongside these, handling file scanning, processing, and batching.

As of April 2026, Max has strong views on which vector database to choose. Weaviate is his default for cloud deployments. Pinecone is the pick for hosted solutions. OpenSearch covers self-hosted cloud. Milvus handles both local and cloud. Qdrant is his go-to for local use because its Rust implementation is low-memory and fast.

On top of the agent layer, we have the observability layer implemented using Opik and OpenTelemetry, along with an evaluation harness with four LLM judges. The evaluation harness is itself pluggable between a local evaluator and Opik.

The configuration is the source of truth for the whole stack. A config.yaml file holds the non-secret details of the vector database, agent, embedding model, and LLM, while secrets are loaded from a .env file. Check all the configs here.

Let us trace a query through the system end-to-end using a concrete example: a user asks about Leica Noctilux lens auctions. The flow unfolds across nine hops, each an Opik span. First, the user submits the natural-language query to the REPL, which immediately starts monitoring the trace with Opik.

The QueryAgent then validates and classifies the intent, passing control to the PlanningAgent to generate an execution plan. Next, the VectorDB adapter performs a semantic search to retrieve relevant documents. The ContextBuilder filters, deduplicates, and sorts these results before handing them to the RAGAgent, which generates the final answer with citations. Finally, the REPL ends the Opik trace, in which every step emits a span containing details such as costs, latency and input/output.

Why Building in Go and Not TypeScript or Python

Most popular agentic CLIs / REPLs, such as Claude Code or OpenCode, are built in TypeScript. But Max, as a former Node.js board chairperson, strongly suggests just going with Go or Rust if memory constraints are a concern.

Why? Because Go apps ship as single binaries. Simple. Beautiful. It runs everywhere.

On-premise customers cannot rely on npm, uv, or JVM registries being reachable inside their own networks, and dependency pinning does not fix network isolation. A single compiled binary sidesteps the entire problem. Go’s track record as the language of Kubernetes is the existing proof that this trade-off works for infrastructure tooling, which is exactly the category Weave CLI sits in. Max himself spent 10 years writing Go on those systems.

Max’s second argument is that language choice matters less than it used to, because AI coding assistants lower the learning-curve barrier across the board.

“Most people don’t write code anymore.” — Max

Still... Max’s newer project (ClawMax.ai) is mostly in TypeScript because it is the best tool for the job, not because he switched allegiances.

“The stack decision has to be what your system wants, not what the herd is doing.” — Max

Next, we zoom into the layers doing the heavy lifting, the ingestion pipeline and the unified VDB interface.

Supporting 11 Vector Databases

The vector database layer is where the ingestion pipeline meets the unified VDB interface. To see how they work together, we’ll trace Max’s Leica Noctilux auction catalog through the system one step at a time.

Each document in the catalog is a single lens listing. It contains a photo of the Noctilux, a short caption with the model number and condition, a price, and a few lines of provenance. The text is sparse. Most of the signal sits in the image itself, and the caption is just enough to disambiguate one Noctilux from another. That sparseness drives a multi-modal ingestion decision up front. The image and the surrounding caption are embedded into two separate collections, one keyed on image vectors and one on caption text vectors. At query time, the auction agent fans out to both collections and merges the results through the ContextBuilder.

Before the actual ingestion, we run a FileScanner that walks the 426 listing files on disk, applying glob matching, exclusion filters, and SHA256 deduplication (src/pkg/pipeline/scanner.go). Re-running ingestion on the same directory skips unchanged documents, making this step fully idempotent and computationally cheap.

The DocumentProcessor extracts text and images from each listing (src/pkg/pipeline/processor.go). For the Leica dataset, the PDF extractor pulls the caption text, and OCR runs on the lens photo to catch any model number printed on the barrel. This step is idempotent but computationally expensive due to PDF parsing and OCR, and it fails if the document format is unsupported. Next, the ChunkingAgent dynamically selects the best chunking strategy for each document.

Next, we move to embedding (src/pkg/embeddings/model_registry.go). In the Leica flow, caption text flows through the text embedder, and image descriptors flow through a separate image embedding model. Raw images larger than per-backend limits (Milvus caps fields at 65KB) get offloaded to S3/MinIO, leaving only a URL in the VDB payload. The default option is to use OpenAI’s embedding model, which is highly expensive in compute and API costs and can fail if you hit rate limits. When scaling, you can use open-source embeddings via Ollama. They run locally with no API key.

The BatchWriter processes documents with durability, such as checkpoint and resume functionality. For example, when ingesting data at scale, you often have network I/O failures or database connection drops. Through checkpointing, we ensure the state is idempotent. Batch checkpointing is the difference between a short retry and a multi-hour rebuild.

“You have to recompute everything from scratch, which is crazy.” — Max

The VectorDBClient Interface sits at the core of the adapter pattern (src/pkg/vectordb/interfaces.go) used to support the 11 databases. The project started with Weaviate. Milvus was surprisingly similar. Qdrant was also very similar. MongoDB was a different beast, but the interface still fit.

“The biggest surprise was PGVector.” — Max

PGVector is the most incompatible on paper. Postgres is a relational database with its own migrations. Yet the unified interface fits.

The pipeline ends at any of the eleven vector databases (src/pkg/vectordb/factory.go), emitting a final vectordb.adapter span. The 426 Leica listings are split into roughly 426 caption vectors in one collection and 426 image vectors in a parallel collection, both sharing listing IDs as the cross-reference key.

These steps cover every component any production ingestion pipeline needs, and Weave CLI ensures each one is swappable by configuration (src/pkg/stack/ingest.go): the FileScanner, the DocumentProcessor, the ChunkingAgent, the embedding provider, the BatchWriter, the VectorDBClient interface, and the concrete VDB adapter.

During retrieval, when a user asks weave query "summarise the 2024 auction catalogue", the QueryAgent classifies the intent, the PlanningAgent decides to hit both Leica collections, and the VectorDB adapter runs a semantic search on each. The ContextBuilder then merges the image-collection hits with the caption-collection hits, deduplicates by listing ID, sorts by relevance score, and extracts content in priority order (caption text first, image metadata second, URL fallback last) into a single prompt for the RAGAgent.

The ingestion pipeline and VDB interface are the skeleton of Weave CLI. The agent layer is what makes it feel like Claude Code for vector databases.

Zooming into the REPL

Weave CLI provides a Claude-Code-like experience for vector databases, which, at its core, is a Read-Eval-Print Loop (REPL) environment hooked up to multiple agents.

Weave CLI ships with 12 built-in agents that you configure via YAML. Three of them are user-facing:

1. Precise QA — asks a question and answers it, and says it cannot answer when it lacks information. Zero hallucination tolerance.

2. RAG — finds the closest chunks and generates an answer over them. This is the default.

3. Summarize — produces a short summary of retrieved chunks.

💡 The beauty is that you can add or modify them as you please.

The next eight agents power the Claude-Code-like orchestration loop: the QueryAgent for intent classification, the PlanningAgent for the execution plan, the WeaveAgent for tool execution with retries, the BashAgent for safe execution, the RAGAgent that the RAG persona dispatches to, the OutputAgent to format progress, the ReportAgent to generate operation reports, and the EvalAgent to track metrics.

The final two are domain helpers used during ingestion: the ChunkingAgent and the SchemaAgent.

Similar to the vector database layer, all the agents implement the same interface:

Let’s tie everything together. When you ask a query, the QueryAgent classifies intent and acts as a router. The PlanningAgent generates a plan of CLI commands. The BashAgent executes them and pipes the output through a command-line JSON processor for filtering. The OutputAgent formats the result. This is the Claude-Code-like loop in action.

The cherry on top is that the Weave CLI capabilities are also exposed as a Model Context Protocol (MCP) server. Thus, instead of using the Weave CLI directly, you can leverage its full functionality through your harness of choice (Claude Code, Codex, etc.).

Twelve agents, eleven databases, five embedding providers, and multiple chunking strategies create a lot of surface area. Opik is what makes the whole thing observable when something breaks.

Monitoring the System

With so many moving parts, you need to know the system is working. Opik is how Weave CLI answers that question: it traces every LLM call, every agent step, and every database write as an OpenTelemetry span.

“Using Opik to tell me how many LLM calls, tokens, and cost per query.” — Max

During development, Max tracked a bug in which documents appeared to be ingested but were never persisted to Milvus. The Opik trace waterfall showed the database flush operations were silently timing out.

💡 If you want to try it out, you can create an account for free on Opik’s managed platform here for 25k spans/month.

The fix was adding dedicated timeout contexts per collection. Without the trace, this would have been a multi-day hunt through logs.

The integration provides cost and latency visibility per trace. You see tokens and dollars per query without writing custom logging. It provides a latency breakdown.

Finally, it provides error visibility to make silent failures loud.

How hard was it to integrate Opik into Weave CLI?

“It’s a very straightforward integration — I pass all queries to the LLMs through Opik via OpenTelemetry, and then I query Opik to aggregate cost from the start of the command to the end.” — Max

Every step in the ingestion and retrieval data flows emits a span (src/pkg/llm/opik.go), which are aggregated under traces containing all the steps between a user request/response.

It includes the query, the LLM reasoning, the tool calls, and the final response. The executor initializes Opik tracing here (src/pkg/executor/executor.go).

Monitoring helps you debug your system. Evaluation moves everything forward, allowing you to quantify your application’s performance.

Evaluating the Default Setup

How do you know your agent is actually better after you swap an embedding model, a vector database or your chunking strategy? You need a good evaluation practice.

“My customers always have five or six questions they ask every release to sanity-check the system. They know what to expect. So I took their QA questions and made them the baseline eval dataset.” — Max

Evaluation datasets come from real user behavior anchored in your business use case, not from standardized, generic benchmarks. If you do not have users yet, you should compile a small set of sanity questions a domain expert would actually ask.

How does this work in Weave CLI?

You start by defining an evaluation dataset in YAML format. This includes the query, expected answer, expected citations, and a minimum relevance score.

Here is the full baseline.yaml file. Or this is how it looks in Opik:

Then you pick an evaluator harness that includes a set of metrics to evaluate against. This harness is itself pluggable: you pick between a local evaluator and Opik (src/pkg/evaluation/provider.go).

We use two families of evaluators. Rule-based evaluators use regular expressions, exact matches, and citation presence (src/pkg/evaluation/custom_evaluator.go) to compute metrics such as CitationMatching for the RAG agent.

They are fast, deterministic, and free. You use them for structural checks.

The second family uses an LLM as a judge. Weave CLI ships four of these judges (src/pkg/evaluation/provider_opik.go). They evaluate Accuracy, Faithfulness, Hallucination, and Context Relevance.

They are slower and cost tokens. You use them for semantic quality.

“The hallucination, citation, and accuracy metrics are all from Opik’s library — I ported them to Golang.” — Max

Then, you run the evaluation command against a chosen agent. Finally, you compare the result of the experiment with the previous run. Each pair of agent and dataset is one experiment (src/cmd/eval/run.go).

The --use-opik flag ships every trace and evaluation result to Opik (src/pkg/evaluation/runner.go). Once in Opik, you get dataset management and experiment comparison.

Scoring every run forces a decision on which agent to ship. Benchmarking on top of your custom datasets provides a structured way to choose a parameter, such as your chunking strategy or top k results, without guessing.

Benchmarking and Optimizing the System

An experiment is a single parameterized run over an agent, dataset, embedding, chunking strategy, database, and judge. A benchmark is a structured set of experiments.

You hold most variables constant to isolate the effect of one. Benchmarking is how you turn random runs into a parameter- and prompt-search problem. This is often known as the optimization flywheel.

“That’s the reason I created Weave CLI. Because this is tedious, but also error-prone.” — Max

Every benchmark is one configuration typo away from drawing the wrong conclusion. Disciplined benchmarking catches that error.

Experiment metadata guarantees reproducibility. Every experiment records the database, embedding model, chunking strategy, dataset, and everything else required to reproduce it. That’s usually the whole config.

Opik tracks this out of the box. Without it, a benchmark from four weeks ago is useless.

When working on RAG systems, the optimization flywheel involves resetting the database, re-ingesting data with new parameters, re-evaluating, and comparing on your metrics of choice.

“Benchmark is comparing multiple agents side by side. Same dataset, different agents — and each (agent, dataset) combination is its own experiment, you can compare later with its metadata.” — Max

You fix a baseline dataset and hold it constant. You vary one axis, typically the agent. You score against multiple metrics.

Each pair of agent and dataset is one Opik experiment. You compare them side-by-side to spot regressions and unexpected wins.

You trigger this loop via the command line with weave eval run --dataset baseline --agents precise-qa,rag,summarize --use-opik. Every subsequent benchmark streams into the same Opik project.

Max ran this loop for his Leica auction customer. He held the dataset and agent constant.

He varied only the embedding provider. He tested OpenAI against sentence-transformers. The open-source model won on quality by 11 percent.

It was 240 times faster for re-embedding. The vectors were 50 percent smaller, and the cost was zero.

This is a counterintuitive outcome. Without a structured benchmark, Max would have defaulted to OpenAI and been wrong.

How to Keep the Flywheel Under Control?

This optimization process involves running your ingestion and retrieval hundreds of times. Which can get costly fast. Super fast. The ingestion checkpointing makes it affordable.

Still, you should optimize your system in order of cheapest-to-change, biggest-win-first [8]. First, tune retrieval parameters like top-K. They are free to change and often provide the biggest wins.

Second, tune the embedding model. It is the cheapest component to swap and has a huge impact. Third, tune the chunking strategy. It requires re-ingestion but offers moderate quality gains.

Finally, tune the vector database. It has the highest switching cost and usually the smallest difference in quality.

The optimization flywheel effectively isolates variables, but it remains a manual process today.

The good news is that Weave CLI is heading toward full automated hyperparameter optimization across databases, embeddings, and chunking strategies. Just imagine. You will launch it before the weekend, and it will return on Monday with the best configuration for your dataset.

You can reproduce this benchmark step by step on your own stack by following this doc.

Watch our full interview on YouTube for all the 3am stories ↓

Final Thoughts

Looking back, what was the hardest thing to implement, and what surprised you the most while building weave-cli? — Paul

The hardest part was designing a unified VectorDBClient that felt natural across 11 providers with wildly different APIs. The adapter pattern was the insight that made it work.

The biggest surprise was benchmarking OSS embeddings against OpenAI on the client’s data and finding them 11% higher quality, 240x faster, and free. A call we’d never have made without evals in place.

If you had to rebuild Weave CLI from scratch, at what point would you introduce monitoring and evaluation? Would you do it earlier, later, or at the same time? — Paul

I’d introduce monitoring from day one. Having Opik traces during the early vector DB work would have immediately surfaced issues such as the silent Milvus persistence failures, which we debugged manually. As for evals, I’d keep at the same stage (after the core RAG pipeline was functional), but I’d design the harness interface up front for citation tracking and confidence scoring.

Opik was easy to integrate and was key to getting the client dashboard working, since I could just run experiments and use evaluations and tracing to decide on the best options for the client.

Now, your next practical step is to experiment with Weave CLI on a real problem. Point it at 100 documents you want to do RAG on, ingest everything into two collections with two different embedding providers, and run the benchmark against the baseline evaluation dataset.

You can follow the step-by-step tutorial from here

But here is what I’m wondering:

While building your latest RAG system, what was your strategy to find the right parameters, such as the embedding model, chunking or retrieval strategies?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

Thanks again to Opik for sponsoring this case study and keeping it free!

If you want to monitor, evaluate and optimize your AI workflows and agents:

Try Opik for free

References

1. Chroma. (n.d.). Evaluating Chunking Strategies for Retrieval. Chroma. https://research.trychroma.com/evaluating-chunking

2. OpenTelemetry. (n.d.). Traces & Spans specification. OpenTelemetry. https://opentelemetry.io/docs/concepts/signals/traces/

3. Husain, H. (n.d.). Creating a LLM-as-a-Judge That Drives Business Results. Hamel Husain. https://hamel.dev/blog/posts/llm-judge/

4. Husain, H. (n.d.). Escaping POC Purgatory: Evaluation-Driven Development for AI. Hamel Husain. https://hamel.dev/blog/posts/evals/

5. Liu, J. (2025, May 19). There Are Only 6 RAG Evals. Jason Liu. https://jxnl.co/writing/2025/05/19/there-are-only-6-rag-evals/

6. Comet. (n.d.). Opik — LLM observability & evaluation platform. GitHub. https://github.com/comet-ml/opik

7. Yan, E. (2024, August 18). Evaluating the Effectiveness of LLM Evaluators (LLM-as-Judge). Eugene Yan. https://eugeneyan.com/writing/llm-evaluators/

8. Liu, J. (2024, February 28). Levels of Complexity: RAG Applications. Jason Liu. https://jxnl.co/writing/2024/02/28/levels-of-complexity-rag-applications/

Images

If not otherwise stated, all images are created by the author.

When building Brown, my writing assistant, I designed five specialized LLM nodes. One handled the introduction, another wrote sections, and others managed the conclusion, title, and editing. It became complicated, slow, and expensive.

I eventually collapsed the system into two agents: a writer and a reviewer operating in a loop. The simpler version performed better. The model retained the full context, and verification became a simple review step rather than a massive orchestration problem.

Most AI teams hit this exact wall. Developers spend more time babysitting AI than engineering, copying error logs and re-prompting models. The real bottleneck is the human.

Three failure modes explain why.

First, context rot. In long AI conversations, the context window becomes a junk drawer. Every failed attempt piles up until the sliding window drops the original specification. The model slides into a “dumb zone” where it hallucinates and forgets its goals. Traditional fixes like summarizing break down over dozens of reasoning rounds.

Second, premature exit. AI agents declare victory too early. Anthropic’s research notes that agents usually look around, see that progress has been made, and declare the job done [1]. Standard ReAct loops inherit the flaw.

Third, single-pass fragility. One prompt, one context, one shot. When it fails, the failure is chaotic. Jumping to multi-agent orchestration introduces distributed systems nightmares.

Ralph loops break the cycle by making “try again with fresh eyes” the default. Named after Ralph Wiggum from The Simpsons, the pattern wipes the conversation, reloads the full specification fresh each iteration, and uses the filesystem and git as the memory layer.

They remove the AI’s ability to grade its own work, using objective signals such as passing tests or linters to call the job done. Boris Cherny, creator of Claude Code, states that giving Claude a way to verify its work increases quality two to three times [2].

One model. One loop. One verification signal. Failure becomes predictable, the loop catches errors and re-prompts automatically, creating a relatively deterministic feedback loop that will 10x the quality of the agent.

Now, let’s look at what Ralph loops are and when you can actually use them in practice.

What Ralph Loops Are and How They Work

Geoffrey Huntley named the pattern after Ralph Wiggum from The Simpsons, noting the character tries the same thing over and over until it works. Huntley’s motto captures the philosophy: the technique is deterministically bad in an undeterministic world. The simplest implementation is a bash while-true loop that pipes a prompt file into the agent forever, acting as a continuous harness pattern [3], [4].

As Einstein reportedly said: “Insanity is doing the same thing over and over again and expecting different results.” Well... I am sure he didn’t predict the rise of Claude Code, because that’s exactly what Ralph loops are all about.

Models are stochastic, strong at reading large contexts but imperfect on first pass. Re-running the same instruction forces self-review. The first iteration produces good but flawed output.

During the second pass, the model spots what it missed and refactors. The third iteration handles cleanup. Huntley delivered a minimum viable product quoted at 50,000 for just 297 in tokens using a single Ralph loop: a 170x cost reduction over the human estimate [3].

You can run these loops in two modes. Shared context keeps the session alive for explicit self-review. Fresh context starts a new session each iteration, removing confirmation bias.

The model sees only the repository and the skill file.

Nowadays, it’s common to replace brittle n8n workflows with a single Claude Code skill. This becomes even more powerful when running in a Ralph loop, especially because at the end of each run, you can take the signal and tell the model to update the skill with anything it should have done differently.

The skill evolves and quality improves automatically [5].

This self-improving mechanism reduces manual prompt tuning when applied to specific, repetitive engineering tasks.

Three Real-World Use Cases

In practice, Ralph loops don’t have a clear implementation pattern. They are more of an intuitive strategy you can get creative with. Thus, you have multiple ways of implementing them.

From my experience with Claude, you have three options for running Ralph loops, from highest abstraction to lowest:

• /ralph-loop plugin — the fastest path. Install it, run /ralph-loop in your session, and it manages the cycle for you.

• /loop command — Claude Code’s built-in scheduler. /loop every 1 minute /your-skill fires the skill on a schedule [6].

• while true bash loop — the most primitive form. A one-liner that pipes a prompt file into the agent and restarts it forever.

Because Claude Code keeps state through the files it’s working on, it retains context from the failed attempt and reads its own git diffs. Each iteration learns from the last.

Implementing a ticket backlog with test-driven development

You can set up a ticket folder with numbered text files. Run a while-true loop that tells Claude to implement the next most important ticket using Test-Driven Development (TDD). The model writes tests first, writes the code, commits the changes, and moves on.

Claude reads all tickets, skips completed ones, picks the next priority, implements it, marks it done, and commits. No dependency graph is needed because the model decides the ordering on the fly. One dumb loop acts like a relentless single-threaded engineer working through the backlog.

For example, set up a doc/tickets folder with numbered tickets (001, 002, 003...). Each describes a feature or fix. Then run:

while true; do
  claude "implement the next most important ticket using TDD principles from doc/tickets. commit when done"
done

Or use Claude Code’s built-in loop:

/loop every 1 minute
build the next ticket from doc/tickets using TDD, run tests, commit when done

Adding test coverage

You can set a concrete goal to raise coverage from 16 percent to 95 percent. The loop reads coverage metrics, writes tests for uncovered functions, runs the suite, identifies gaps, and iterates.

The coverage report provides the objective backpressure. The loop does not stop until the numbers validate success. Each iteration chips away at untested code paths until the threshold is met.

The implementation is as easy as:

while true; do
  claude "analyze coverage gaps, write tests for uncovered functions, run the test suite, fix failures. stop when coverage exceeds 95%"
done

Framework and dependency migrations

Migrations require crisp completion criteria. Upgrading React v16 to v19, Next.js 14 to 15, or migrating Jest to Vitest demands a clean build and passing tests. The agent swaps syntax, updates dependencies, and runs build commands.

It uses compiler errors and failing tests as feedback. Each cycle fixes a batch of errors until the toolchain confirms the code is clean. Deterministic verification signals make framework migrations the perfect Ralph loop candidate.

These are three concrete starting points. Before you wire the first one up, there is one honest limit you should know.

What’s Next

Ralph loops are the starting point. Once comfortable, add self-improving skills that update their instructions after each run, wire stop hooks for objective quality gates to avoid infinite loops, and connect to external systems like Linear or GitHub Issues so the loop reacts to new work automatically.

The pattern scales further than it looks. OpenAI’s Codex team shipped one million lines of code across 1,500 pull requests with zero human-written code using what they call a “Ralph Wiggum Loop” [7].

These loops are safe when repo-contained and the toolchain acts as the judge. They get dangerous with irreversible side effects outside the repo. Alexey Grigorev learned this when a Claude Code agent ran terraform destroy on DataTalks.Club’s production infrastructure, wiping the database, VPC, and all automated snapshots — two and a half years of data gone in one iteration. If your loop can destroy shared state, review every plan manually [8].

What is the first piece of work in your repo you would trust a Ralph loop with? You could choose a TDD backlog, a coverage ramp, a framework migration or what else?

Click the button below and tell me. I read every response.

Leave a comment

Enjoyed the article? The most sincere compliment is to restack this for your readers.

Share

References

1. Anthropic. (n.d.). Effective Harnesses for Long-Running Agents. Anthropic. https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents

2. Cherny, B. (n.d.). I’m Boris and I Created Claude Code. X. https://x.com/bcherny/status/2007179832300581177

1. Huntley, G. (n.d.). Ralph Wiggum as a “software engineer”. Geoffrey Huntley. https://ghuntley.com/ralph/

2. LangChain. (n.d.). The Anatomy of an Agent Harness. LangChain Blog. https://blog.langchain.com/the-anatomy-of-an-agent-harness/

3. Parsons, C. (n.d.). Ralph Loops: Build Dumb AI Loops That Ship. AI Engineer. https://read.readwise.io/read/01kp5bgy8b07y256ythvkz2tt7

4. Anthropic. (n.d.). Claude Code: Best Practices for Agentic Coding. Anthropic. https://www.anthropic.com/engineering/claude-code-best-practices

5. Lopopolo, R. (n.d.). Harness engineering: leveraging Codex in an agent-first world. OpenAI. https://openai.com/index/harness-engineering-codex/

6. Grigorev, A. (n.d.). How I Dropped Our Production Database and Now Pay 10% More for AWS. Alexey Grigorev. https://alexeygrigorev.com/posts/dropped-production-database/

Images

If not otherwise stated, all images are created by the author.