Escaping POC Purgatory: Evaluation-Driven Development for AI Systems
A new software development life cycle for LLMs
Paul: Today, the scene is owned by two people who see applied AI from every angle: Hugo and Stefan Krawczyk.
You have Hugo, the brilliant mind, teaching LLM systems to engineers at Netflix and Meta. Joining him is Stefan, the CEO of DAGWorks, who has spent over a decade leading ML teams that build real-world systems.
They’ve put together a course on the LLM software development lifecycle, focusing on everything from retrieval and evaluation to agent design, and all the intermediate steps in between.
Enough chitchat. Let’s get into it 👀 ↓
Escaping from Proof of Concept (POC) Purgatory
The demo is the easy part.
Building AI-powered applications today can feel like purgatory. Someone hacks together a quick demo with ChatGPT and Langchain, leadership gets excited: “We can answer any question about our docs!” But then… the system is inconsistent, slow, hallucinating: and that amazing demo starts collecting digital dust. We call this “POC Purgatory”, that frustrating limbo where you’ve built something cool but can’t quite turn it into something real.
We’ve seen this across dozens of companies, and the teams that break out of this trap all adopt some version of Evaluation-Driven Development (EDD), where testing, monitoring, and evaluation drive every decision from the start.
The truth is, we’re in the earliest days of understanding how to build robust LLM applications. Most teams approach this like traditional software development, but quickly discover it’s a fundamentally different beast. Check out the graph below: see how excitement for traditional software builds steadily while GenAI starts with a flashy demo and then hits a wall of challenges?
What makes LLM applications so different? Two big things:
They bring the messiness of the real world into your system through unstructured data;
They’re fundamentally non-deterministic… we call it the “flip-floppy” nature of LLMs: same input, different outputs. What’s worse? inputs are rarely exactly the same! Tiny changes in user queries, phrasing, or surrounding context can lead to wildly different results.
This creates a whole new set of challenges that traditional software development approaches simply weren’t designed to handle. When your system is both ingesting messy real-world data AND producing non-deterministic outputs, you need a different approach.
The way out? Evaluation-driven development, a systematic approach where continuous testing and assessment guide every stage of your LLM application’s lifecycle. This isn’t anything new. People have been building data products and machine learning products for the past couple of decades. The best practices in those fields have always centered around rigorous evaluation cycles. We’re simply adapting and extending these proven approaches to address the unique challenges of LLMs and AI.
We’ve been working with dozens of companies building LLM applications, and we’ve noticed patterns in what works and what doesn’t. In this article, we’re going to share an emerging SDLC for AI applications that can help you escape POC purgatory. We won’t be prescribing specific tools or frameworks (those will change every few months anyway), but rather the enduring principles that can guide effective development regardless of which tech stack you choose.
Throughout this article, we’ll explore real-world examples of AI application development, and then consolidate what we’ve learned into a set of first principles, covering areas like non-determinism, evaluation approaches, and iteration cycles, that can guide your work regardless of which models or frameworks you choose.
Focus on Principles, not Frameworks
A lot of people ask us: What tools should I use? Which multi-agent frameworks? Should I be using multi-turn conversations or LLM-as-judge?
Of course, we have opinions on all of these, but we think those aren’t the most useful questions to ask right now. We’re betting that lots of tools, frameworks, and many techniques will disappear or change, but certain principles in building AI-powered applications will remain.
We’re also betting that this will be a time when software development flourishes. With the advent of Generative AI, there’ll be significant opportunities for product managers, designers, executives, and more traditional software engineers to contribute to and build AI-powered software. One of the great aspects of the AI Age is that more people will be able to build software.
We’ve been working with dozens of companies building AI-powered applications and have started to see clear patterns in what works. We’ve taught this SDLC in a live course with engineers from companies like Netflix, Meta, and the US Air Force, and recently distilled it into a free 10-email course to help teams apply it in practice.
Is AI-Powered Software Actually That Different from Traditional Software?
When building AI-powered software, the first question is: should my software development lifecycle be any different from a more traditional SDLC, where we build, test, and then deploy?
AI-powered applications introduce more complexity than traditional software in several ways:
Introducing the entropy of the real world into the system through data;
The introduction of non-determinism or stochasticity into the system: the most obvious symptom here is what we call the flip floppy nature of LLMs, that is, you can give an LLM the same input and get two different results;
The cost of iteration: in compute, staff time, and ambiguity around product readiness;
The coordination tax: LLM outputs are often evaluated by non-technical stakeholders (legal, brand, support): not just for functionality, but for tone, appropriateness, and risk. This makes review cycles messier and more subjective than in traditional software or ML.
What breaks your app in production isn’t always what you tested for in dev!
This inherent unpredictability is precisely why evaluation-driven development becomes essential: rather than an afterthought, evaluation becomes the driving force behind every iteration.
Evaluation is the engine, not the afterthought.
The first property is something we saw with data and ML-powered software. What this meant was the emergence of a new stack for ML-powered app development, often referred to as MLOps. It also meant three things:
Software was now exposed to a potentially large amount of messy, real-world data;
ML apps needed to be developed through cycles of experimentation (as we’re no longer able to reason about how they’ll behave, based on software specs);
The skillset and the background of people building the applications were realigned: people who were at home with data and experimentation got involved!
Now with LLMs, AI, and their inherent flip-floppiness, an array of new issues arises:
Non-determinism: how can we build reliable and consistent software using models that are non-deterministic and unpredictable?
Hallucinations and forgetting: how can we build reliable and consistent software using models that both forget and hallucinate?
Evaluation: How do we evaluate such systems, especially when outputs are qualitative, subjective, or hard to benchmark?
Iteration: We know we need to experiment with and iterate on these systems…. How to do so?
Business value: once we have a rubric for evaluating our systems, how do we tie our macro-level business metrics to our micro-level LLM evaluations? This becomes especially difficult when outputs are qualitative, subjective, or context-sensitive, a challenge we saw in MLOps, but one that’s even more pronounced in GenAI systems.
Beyond the technical challenges, these complexities also have real business implications. Hallucinations and inconsistent outputs aren’t just engineering problems. They can erode customer trust, increase support costs, and lead to compliance risks in regulated industries. That’s why integrating evaluation and iteration into the SDLC isn’t just good practice, it’s essential for delivering reliable, high-value AI products.
A Typical Journey in Building AI-Powered Software
In this section, we’ll walk through a real-world example of an LLM-powered application struggling to move beyond the proof-of-concept stage. Along the way, we’ll explore:
Why defining clear user scenarios and understanding how LLM outputs will be used in the product prevents wasted effort and misalignment.
How synthetic data can accelerate iteration before real users interact with the system.
Why early observability (logging & monitoring) is crucial for diagnosing issues.
How structured evaluation methods move teams beyond intuition-driven improvements.
How error analysis and iteration refine both LLM performance and system design.
By the end, you’ll see how this team escaped POC purgatory, not by chasing the perfect model, but by adopting a structured development cycle that turned a promising demo into a real product.
At its core, this case study demonstrates evaluation-driven development in action. Instead of treating evaluation as a final step, we use it to guide every decision from the start, whether choosing tools, iterating on prompts, or refining system behavior. This mindset shift is critical to escaping POC purgatory and building reliable LLM applications.
POC purgatory
Every LLM project starts with excitement. The real challenge is making it useful at scale.
The story doesn’t always start with a business goal. Recently, we helped an EdTech startup build an information-retrieval app. Someone realized they had tons of content a student could query. They hacked together a prototype in ~100 lines of Python using OpenAI and LlamaIndex. Then they slapped on tool use to search the web, saw low retrieval scores, called it an “agent,” and called it a day. Just like that, they landed in POC purgatory, stuck between a flashy demo and working software.
They tried various prompts and models and, based on eyeballing results and “vibes”, decided some were better than others. They also realized that, although LlamaIndex was cool to get this POC out the door, they couldn’t easily figure out what prompt it was throwing to the LLM, what embedding model was being used, the chunking strategy, and so on. So they let go of LlamaIndex for the time being and started using vanilla Python and basic LLM calls. They used some local embeddings and played around with different chunking strategies. Some seemed better than others.
💡 Lessons Learned: POC Purgatory
A prototype isn’t a product—without evaluation, it’s just a cool demo.
Many tools abstract too much—re-implementing key pieces yourself (e.g., in plain Python) often reveals what’s really going on.
“LLM magic” is often just uncontrolled variability—adding structure makes it repeatable.
Evaluating your model with vibes, scenarios, and personas
Before you can evaluate an LLM system, you need to define who it’s for and what success looks like.
They then decided to try to formalize some of these “vibe checks” into an evaluation framework (commonly called a “harness”), which they can use to test different versions of the system. But wait: what do they even want the system to do? Who do they want to use it? Eventually, they want to roll it out to students, but perhaps a first goal would be to roll it out internally.
Vibes are a fine starting point, just don’t stop there.
We asked them:
Who are you building it for?
In what scenarios do you see them using the application?
How will we measure success?
The answers were:
Our students
Any scenario in which a student is looking for information that the corpus of documents can answer
If the student finds the interaction helpful.
The first answer came easily, the second was a bit more challenging, and the team didn’t even seem confident with their 3rd answer. What counts as success depends on who you ask.
We suggested:
They keep the goal of building it for students, but orient first around whether internal staff find it useful before rolling it out to students;
Restricting the first goals of the product to something actually testable, such as giving helpful answers to FAQs about course content, course timelines, and instructors.
Keeping the goal of finding the interaction helpful, but recognizing that this contains a lot of other concerns, such as clarity, concision, tone, and correctness.
So now we have a user persona, several scenarios, and a way to measure success.
💡 Lessons Learned: Evaluating Your Model with Vibes, Scenarios, and Personas
If you don’t define success, you can’t measure it.
User scenarios help prevent pointless optimizations—you’re building for real needs, not just tweaking for better vibes.
Testing with synthetic queries and internal usage helps catch major issues before exposing LLMs to users.
Synthetic data for your LLM flywheel
Why wait for real users to generate data when you can bootstrap testing with synthetic queries?
With traditional, or even ML, software, you’d then usually try to get some people to use your product.. But we can also use synthetic data, starting with a few manually written queries, then using LLMs to generate more based on user personas, to simulate early usage and bootstrap evaluation.
So we did that. We made them generate ~50 queries. To do this, we needed logging, which they already had, and we needed visibility into the traces (prompt and response). There were non-technical SMEs we wanted in the loop.
Also, we’re now trying to develop our eval harness so we need “some form of ground truth”, that is, examples of user queries + helpful responses.
This systematic generation of test cases is a hallmark of evaluation-driven development: creating the feedback mechanisms that drive improvement before real users encounter your system.
💡 Lessons Learned: Synthetic Data for Your LLM Flywheel
Generating test cases in advance speeds up iteration.
Even small-scale synthetic data uncovers system weaknesses early.
Logging isn’t just debugging—it’s the foundation for reproducible improvements.
Looking at your data, error analysis, and rapid iteration
Logging and iteration aren’t just debugging tools, they’re the heart of building reliable LLM apps. You can’t fix what you can’t see.
To build trust with our system, we need to confirm at least some of the responses with our own eyes. So we pulled them up in a spreadsheet and got our SMEs to label responses as “helpful or not” and to also give reasons.
Then we iterated on the prompt and noticed that it did well with course content but not as well with course timelines. Even this basic error analysis allowed us to decide what to prioritize next.
When playing around with the system, I tried a query that many people ask LLMs with Information Retrieval, but few engineers think to handle: “What docs do you have access to?” RAG performs horribly with this most of the time. An easy fix for this involved engineering the system prompt.
Essentially, what we did here was:
Build,
Deploy (to only a handful of internal stakeholders),
Log/Monitor/Observe,
Evaluate & error analysis, and
Iterate.
Now it didn’t involve rolling out to external users, it didn’t involve frameworks, it didn’t even involve a robust eval harness yet, and the system changes involved only prompt engineering. It involved a lot of looking at your data*! We only knew how to change the prompts for the biggest effects by performing our error analysis.
*For more on this, check out Hugo Bowne-Anderson and Hamel Husain’s Vanishing Gradients livestream and podcast episode.
What we see here, though, is the emergence of the first iterations of the LLM SDLC: we’re not yet changing our embeddings, fine-tuning, or business logic, we’re not using unit tests, CI/CD, or even a serious evaluation framework, but we’re building, deploying, monitoring, evaluating, and iterating!
First eval harness
Evaluation must move beyond ‘vibes’: a structured, reproducible harness lets you compare changes reliably.
In order to build our first eval harness, we needed some ground truth, that is, a user query and an acceptable response with sources.
To do this, we either needed SMEs to generate acceptable responses + sources from user queries or have our AI system generate them and an SME to accept or reject them. We chose the latter.
So we generated 100 user interactions and used the accepted ones as our test set for our evaluation harness. We tested both retrieval quality (i.e., how well the system fetched relevant document, measured with metrics like precision and recall), semantic similarity of response, cost, and latency, in addition to performing heuristics checks, such as length constraints, hedging vs overconfidence, and hallucination detection.
We then used thresholding of the above to either accept or reject a response. However, looking at why a response was rejected helped us iterate quickly:
🚨 Low similarity to accepted response: Reviewer checks if the response is actually bad or just phrased differently.
🔍 Wrong document retrieval: Debug chunking strategy, retrieval method.
⚠️ Hallucination risk: Add stronger grounding in retrieval or prompt modifications.
🏎️ Slow response / high cost: Optimize model usage or retrieval efficiency.
There are many parts of the pipeline one can focus on, and error analysis will help you prioritize. Depending on your use case, this might mean evaluating RAG components (e.g. chunking or OCR quality), basic tool use (e.g. calling an API for calculations), or even agentic patterns (e.g. multi-step workflows with tool selection). For example, if you’re building a document QA tool, upgrading from basic OCR to AI-powered extraction (think Mistral OCR) might give the biggest lift on your system!
In the initial iterations, we also needed to refine our eval harness by examining its outputs and adjusting our thresholding accordingly.
And just like that, the eval harness becomes more than just a QA tool: it becomes the operating system for iteration!
💡 Lessons Learned: Looking at Your Data, Error Analysis, and Rapid Iteration + First Eval Harness
Evaluation must be reproducible—without structure, you’re just guessing.
A simple eval harness is better than none—start small, refine often.
First Principles of LLM-Powered Application Design
What we’ve seen here is the emergence of an SDLC distinct from the traditional SDLC and similar to the ML SDLC, with the added nuances of now needing to deal with non-determinism and masses of natural language data.
The key shift in this SDLC is that evaluation isn’t a final step, it’s an ongoing process that informs every design decision. Unlike traditional software development, where functionality is often validated after the fact with tests or metrics, AI systems require evaluation and monitoring to be built in from the start. In fact, acceptance criteria for AI applications must explicitly include evaluation and monitoring. This is often surprising to engineers coming from traditional software or data infrastructure backgrounds, who may not be used to thinking about validation plans until after the code is written.
We’ve also seen the emergence of the First Principles for Generative AI and LLM software development. These principles are
We’re working with API calls: these have inputs (prompts) and outputs (responses); we can add memory, context, tool use, and structured outputs using both the system and user prompts; we can turn knobs, such as temperature and top p;
LLM calls are non-deterministic: the same inputs can result in drastically different outputs ← this is an issue for software!
Logging/monitoring/tracing: You need to capture your data;
Evaluation: You need to look at your data and results and quantify performance (a combination of domain expertise and binary classification)
Iteration: Iterate quickly using prompt engineering, embeddings, tool use, fine-tuning, business logic, and more!
As a result of this new SDLC, we get methods to help us through the challenges we’ve identified:
Non-determinism: log inputs and outputs, evaluate logs, iterate on prompts and context, use API knobs to reduce variance of outputs;
Hallucinations and forgetting:
Log inputs and outputs in dev and prod;
Use domain-specific expertise to evaluate output in dev and prod;
Build systems and processes to help automate assessment, such as unit tests, datasets, and product feedback hooks.
Evaluation: Same as above.
Iteration: Build an SDLC that allows you to rapidly Build → Deploy → Monitor → Evaluate → Iterate;
Business value: Align outputs with business metrics and optimize workflows to achieve measurable ROI.
An astute and thoughtful reader may point out that the SDLC for traditional software is also somewhat circular: Nothing’s ever finished; you release 1.0 and immediately start on 1.1.
We don’t disagree with this, but we’d add that, with traditional software, each version completes a clearly defined, stable development cycle. Iterations produce predictable, discrete releases.
By contrast:
ML-powered software introduces uncertainty due to real-world entropy (data drift, model drift), making testing probabilistic rather than deterministic.
LLM-powered software amplifies this uncertainty further. It isn’t just natural language that’s tricky; it’s the “flip-floppy” non-deterministic behavior, where the same input can produce significantly different outputs each time.
Reliability isn’t just a technical concern: it’s a business one. Flaky or inconsistent LLM behavior erodes user trust, increases support costs, and makes products harder to maintain. Teams need to ask: What’s our business tolerance for that unpredictability, and what kind of evaluation or QA system will help us stay ahead of it?
This unpredictability demands continuous monitoring, iterative prompt engineering, maybe even fine-tuning, and frequent updates just to maintain basic reliability.
So traditional software is iterative but discrete and stable, while LLM-powered software is genuinely continuous and inherently unstable without constant attention, it’s more of a continuous limit than distinct version cycles.
Getting out of POC purgatory isn’t about chasing the latest tools or frameworks, it’s about committing to evaluation-driven development through an SDLC that makes AI systems observable, testable, and improvable. Teams that embrace this shift will be the ones that turn promising demos into real, production-ready AI products.
The AI age is here, and more people than ever have the ability to build. The question isn’t whether you can launch an LLM app. It’s whether you can build one that lasts, and drives real business value.
Want to go deeper? We created a free 10-email course that walks through how to apply these principles — from user scenarios and logging to evaluation harnesses and production testing. And if you’re ready to get hands-on with guided projects and community support, the next cohort of our Maven course kicks off on November 3rd (use code PAULxMAVEN for $100 off).
Many thanks to Shreya Shankar, Bryan Bischof, Nathan Danielsen, and Ravin Kumar for their valuable and critical feedback on drafts of this essay along the way.
What’s your take on today’s topic? Do you agree, disagree, or is there something I missed?
If you enjoyed this article, the ultimate compliment is to share our work.
Copyrights: A version of this article was initially published on O’Reilly Radar by Hugo Bowne-Anderson and Stefan Krawczyk.
Images
If not otherwise stated, all images are created by the author.
 | A guest post by
|
 | A guest post by
|
Thanks for yet another wonderful collaboration, Paul!
I hope that it helps your audience build more reliable AI-powered software :)
This piece really resonated with me. The 'POC Purgatory' is such an accurate description; it’s a proplem I've seen too often. Your emphasis on Evalution-Driven Development is spot on. It truly feels like we're still figuring out the engineering principles for robust LLM apps. A fantastic read.