The sixth lesson of the open-source course Building Your Second Brain AI Assistant Using Agents, LLMs and RAG — a free course that will teach you how to architect and build a personal AI research assistant that talks to your digital resources.

A journey where you will have the chance to learn to implement an LLM application using agents, advanced Retrieval-augmented generation (RAG), fine-tuning, LLMOps, and AI systems techniques.

Lessons:

Lesson 1: Build your Second Brain AI assistant

Lesson 2: Data pipelines for AI assistants

Lesson 3: From noisy docs to fine-tuning datasets

Lesson 4: Fine-tune and deploy open-source LLMs

Lesson 5: Build RAG pipelines that actually work

Lesson 6: LLMOps for production agentic RAG

🔗 Learn more about the course and its outline.

LLMOps for production agentic RAG

Welcome to Lesson 6 of Decoding ML’s Building Your Second Brain AI Assistant Using Agents, LLMs and RAG open-source course, where you will learn to architect and build a production-ready Notion-like AI research assistant.

Agents are the latest breakthrough in AI. For the first time in history, we give a machine complete control over its decisions without explicitly telling it. Agents do that through the LLM, the system's brain that interprets the queries and decides what to do next and through the tools that provide access to the external world, such as APIs and databases.

One of the agents' most popular use cases is Agentic RAG, in which agents access a tool that provides them with access to a vector database (or another type of database) to retrieve relevant context dynamically before generating an answer.

Agentic RAG differs from a standard RAG workflow in that the LLM can dynamically choose when it needs context or whether a single query to the database provides enough context.

Agents, relative to workflows, introduce even more randomness into the system. This is a core reason why adding LLMOp best practices such as prompt monitoring and LLM evaluation is a critical step in making your system easy to debug and maintain.

LLMOps and evaluation are critical in any AI system, but they become even more crucial when working with agents!

In previous lessons of the course, we implemented all the offline pipelines that helped us prepare for advanced RAG, such as populating the MongoDB vector index with the proper data from our Second Brain and fine-tuning a summarization open-source small language model (SLM).

In this lesson, we will take the final step to glue everything together by adding an agentic layer on top of the vector index and an observability module on top of the agent to monitor and evaluate it. These elements will be part of our online inference pipelines, which will turn into the Second Brain AI assistant that the user interacts with, as seen in the demo below:

Thus, in this lesson, we will dive into the fundamentals of agentic RAG, exploring how agents powered by LLMs can go beyond traditional retrieval-based workflows to dynamically interact with multiple tools and external systems, such as vector databases.

Next, we will move to our observability pipeline, which evaluates the agents using techniques such as LLM-as-judges and heuristics to ensure they work correctly. We will monitor the prompt traces that power the agents to help us debug and understand what happens under the hood.

While going through this lesson, we will learn the following:

Understand what an agent is, how it differs from workflows, and why it’s useful.
Architect the Agentic RAG module, understanding its components and data flow.
Build and monitor an agentic LLM application using SmolAgents and Opik.
Implement prompt monitoring pipelines to track input/output, latency, and metadata.
Explore RAG evaluation metrics like moderation, hallucination, and answer relevance.
Create custom evaluation metrics, integrating heuristics and LLM judges.
Automate observability, ensuring real-time performance tracking.
Interact with the Second Brain AI assistant via CLI or a beautiful Gradio UI.

Figure 1: The architecture of the Second Brain AI assistant powered by RAG, LLMs and agents.

Let’s get started. Enjoy!

Podcast version of the lesson

Table of contents:

Understanding how LLM-powered agents work
Researching Agentic RAG
Exploring the difference between agents and workflows
Architecting the Agentic RAG module
Understanding how to evaluate an agentic RAG application
Architecting the observability pipeline
Implementing the agentic RAG module
Building the LLM evaluation pipeline
Running the code

1. Understanding how LLM-powered agents work

LLM-powered agents combine a language model, tools, and memory to process information and take action.

They don’t just generate text—they reason, retrieve data, and interact with external systems to complete tasks.

At its core, an agent takes in an input, analyzes what needs to be done, and decides the best way to respond. Instead of working in isolation, it can tap into external tools like APIs, databases, or plugins to enhance its capabilities.

With the reasoning power of LLMs, the agent doesn’t just react—it strategizes. It breaks down the task, plans the necessary steps, and takes action to get the job done efficiently.

Figure 1: The components of an LLM-powered agent

The most popular way to design agents is by using the ReAct framework, which models the agent as follows:

act: the LLM calls specific tools
observe: pass the tool output back to the LLM
reason: the LLM reason about the tool output to decide what to do next (e.g., call another tool or respond directly)

Now, let’s understand how agents and RAG fit together.

2. Researching Agentic RAG

Unlike a traditional RAG setup's linear, step-by-step nature, Agentic RAG puts an agent at the center of decision-making.

Instead of passively retrieving and generating responses, the agent actively directs the process—deciding what to search for, how to refine queries, and when to use external tools, such as SQL, vector, or graph databases.

Figure 2: The single-agent RAG system architecture

For example, instead of querying the vector database just once (what we usually do in a standard RAG workflow), the agent might decide that after its first query, it doesn’t have enough information to provide an answer, making another request to the vector database with a different query.

3. Exploring the difference between agents and workflows

Now that we’ve explored LLM-powered agents and Agentic RAGs, let’s take a step back and look at a broader question: “How do agents differ from workflows?” While both help automate tasks, they operate in fundamentally different ways.

A workflow follows a fixed, predefined sequence—every step is planned in advance, making it reliable but rigid (more similar to classic programming).

In contrast, an agent dynamically decides what to do next based on reasoning, memory, and available tools. Instead of just executing steps, it adapts, learns, and makes decisions on the fly.

Think of a workflow as an assembly line, executing tasks in order, while an agent is like an intelligent assistant, capable of adjusting its approach in real time. This flexibility makes agents powerful for handling unstructured, complex problems that require dynamic decision-making.

Figure 3: Differences between workflows and agents

Therefore, the trade-off between reliability and adaptability is key—workflows offer stability but are rigid, while agents provide flexibility by making dynamic decisions at the cost of consistency.

Now that we understand the basics of working with agents, let’s dive into the architecture of our Second Brain agent.

4. Architecting the Agentic RAG module

When architecting the Agentic RAG module, the goal is to build an intelligent system that efficiently combines retrieval, reasoning, and summarization to generate high-quality responses tailored to user queries.

What’s the interface of the pipeline?

The pipeline takes a user query as input (submitted through the Gradio UI).

The output is a refined answer generated by the agent after reasoning, retrieving relevant context from MongoDB through semantic search, and processing it through the summarization tool.

Offline vs. online pipelines

The Agentic RAG module fundamentally differs from the offline ML pipelines we’ve built in previous lessons.

This module is entirely decoupled from the pipelines in Lessons 1-5. It lives in a separate second-brain-online folder within our repository as its own standalone Python application.

This separation is intentional—by keeping the offline pipelines (feature and training) fully independent from the online inference system, we ensure a clean architectural divide.

As a quick reminder from Lesson 1, offline pipelines are batch pipelines that run on a schedule or trigger. They process input data and store the output artifacts in storage, allowing other pipelines or clients to consume them as needed.

These include the data collection pipeline, ETL pipeline, RAG feature pipeline, dataset generation pipeline, and training pipeline. They operate independently and are decoupled through various storage solutions such as document databases, vector databases, data registries, or model registries.

The Agentic RAG module, on the other hand, belongs to the category of online pipelines. It directly interacts with the user and must remain available at all times. The online pipelines available in this project are the agentic inference pipeline, the summarization inference pipeline, and the observability pipeline.

Unlike offline pipelines, these do not require orchestration and function similarly to RESTful APIs, ensuring minimal latency and efficient responses.

What does the pipeline’s architecture look like?

The Agentic RAG module operates in real time, instantly responding to user queries without redundant processing.

This module's core is an agent-driven system that reasons independently and dynamically invokes tools to handle user queries. They serve as extensions of the LLM model powering the agent, allowing it to perform tasks it wouldn’t efficiently handle on its own without specialized fine-tuning.

Our agent relies on three main components:

The what can I do tool, which helps users understand the usages of the system
The retriever tool that queries MongoDB’s vector index pre-populated during our offline processing
The summarization tool uses a REST API to call a different model specialized in summarizing web documents.

We specifically picked these ones as they are a perfect use case for showing how to use a tool that runs only with Python, one that calls a database, and one that calls an API (three of the most common scenarios).

The agent layer is powered by the SmolAgents framework (by Hugging Face) and orchestrates the reasoning process. A maximum number of steps can be set to ensure the reasoning remains focused and does not take unnecessary iterations to reach a response (avoiding skyrocketing bills).

To provide a seamless user experience, we integrated the agentic inference pipeline with a Gradio UI, making interactions intuitive and accessible. This setup ensures that users can engage with the assistant as naturally as possible, simulating a conversational AI experience.

The interface allows us to track how the agent selects and uses tools during interactions.

For instance, we can see when it calls the MongoDB vector search tool to retrieve relevant data and how it cycles between retrieving information and reasoning before generating a response.

Figure 4: The architecture of the Agentic RAG module

The agentic inference pipeline is designed to handle user queries in real time, orchestrating a seamless data flow from input to response. To understand how information moves through the system, we break down the interaction between the user, the retrieval process, and the summarization mechanism.

When a user submits a query through the Gradio UI, the Agentic Layer, an LLM-powered agent, dynamically determines the most suitable tool to process the request.

If additional context is required, the Retriever Tool fetches relevant information from the MongoDB vector database, extracting the most relevant chunks. This vector database was previously populated through the RAG feature pipeline in Lesson 5, ensuring the system has preprocessed, structured knowledge readily available for retrieval.

The retrieved data is then refined using the Summarization Tool, which enhances clarity before generating the final response. For summarization, we can choose between a custom Summarization Inference Pipeline, which is powered by the Hugging Face model we trained in Lesson 4, or an OpenAI model.

The agent continues reasoning iteratively until it reaches the predefined step limit or it decides it has the final answer, ensuring efficiency while maintaining high response quality.

As a side note, given the simplicity of our use case, the Second Brain AI assistant could have been implemented using a traditional workflow, directly retrieving and responding to queries without an agentic approach.

However, by embracing this modular strategy, we achieve greater scalability and flexibility, allowing the system to integrate new data sources or tools easily in the future.

Now that we understand how the agent works, let’s dig into how we can evaluate it and then into the implementation.

5. Understanding how to evaluate an agentic RAG application

When evaluating an Agentic RAG application, it’s important to distinguish between two primary evaluation approaches: LLM evaluation and Application/RAG evaluation. Each serves a different purpose, and while LLM evaluation assesses the model in isolation, Application/RAG evaluation tests the entire application as a system.

In this case, our primary focus is evaluating the RAG pipeline as a black-box system, assessing how retrieval and reasoning work together to generate the final output.

However, we also provide a brief refresher on key insights from LLM evaluation in Lesson 4 to highlight its role in the broader evaluation process.

LLM evaluation

As a brief reminder, LLM evaluation measures response quality without retrieval. In Lesson 4, we tested this by analyzing the model’s ability to generate answers from its internal knowledge.

Popular methods for LLM evaluation include benchmark-based evaluation (using standardized datasets), heuristic evaluation (ROUGE, BLEU, regex matching, or custom heuristics), semantic-based evaluation (BERT Score), and LLM-as-a-judge, where another LLM evaluates the generated outputs.

Each method has strengths and trade-offs. Benchmark-based evaluation provides standardized comparisons but may not fully capture real-world performance, while heuristic methods may offer quick, interpretable insights but often fail to assess deeper contextual understanding. Additionally, LLM-as-a-judge is flexible and scalable, though it introduces potential biases from the evaluating model itself.

RAG evaluation

Unlike LLM evaluation, which assesses the model’s ability to generate responses from internal knowledge, RAG evaluation focuses on how well the retrieval and generation processes work together.

Evaluating a RAG application requires analyzing how different components interact. We focus on four key dimensions:

User input – The query submitted by the user.
Retrieved context – The passages or documents fetched from the vector database.
Generated output – The final response produced by the LLM based on retrieved information.
Expected output – The ideal or ground-truth answer, if available, for comparison.

By evaluating these dimensions, we can determine whether the retrieved context is relevant, the response is grounded in the retrieved data, and the system generates complete and accurate answers.

As mentioned, we break the process into two key steps to evaluate a RAG application correctly: retrieval and generation.

Since RAG applications rely on retrieving relevant documents before generating responses, retrieval quality plays a critical role in overall performance. If the retrieval step fails, the LLM will either generate incorrect answers or hallucinate information.

To assess retrieval step effectiveness, we can use various ranking-based metrics, including:

NDCG (Normalized Discounted Cumulative Gain) – Measures how well the retrieved documents are ranked, prioritizing the most relevant ones at the top.
MRR (Mean Reciprocal Rank) – Evaluates how early the first relevant document appears in the retrieved results, ensuring high-ranking relevance.

Another option is to visualize the embedding from your vector index (using algorithms such as t-SNE or UMAP) to see if there are any meaningful clusters within your vector space.

On the other hand, during the generation step, you can leverage similar strategies we looked at in the LLM evaluation subsection while considering the context dimension.

LLM application evaluation

For LLM application evaluation, we take a black-box approach, meaning we assess the entire Agentic RAG module rather than isolating individual components.

We evaluate the entire system by analyzing the input, output, and retrieved context instead of separating retrieval and generation into independent evaluations.

This approach allows us to identify system-wide failures and measure how well the retrieved knowledge contributes to generating accurate and relevant responses.

By evaluating the entire module, we can detect common RAG issues, such as hallucinations caused by missing context or low retrieval recall leading to incomplete answers, ensuring the system performs reliably in real-world scenarios.

How many samples do we need to evaluate our LLM app?

Naturally, using too few samples for evaluation can lead to misleading conclusions. For example, 5-10 examples are insufficient for capturing meaningful patterns, while 30-50 examples provide a reasonable starting point for evaluation.

Ideally, a dataset of over 400 samples ensures a more comprehensive assessment, helping to uncover biases and edge cases.

What else should be monitored along the LLM outputs?

Beyond output quality, system performance metrics like latency, throughput, reliability, and costs should be tracked to ensure scalability.

Additionally, business metrics—such as conversion rates, user engagement, or behavior influenced by the assistant—help measure the real-world impact of the LLM application.

Popular evaluation tools

Several tools specialize in RAG and LLM evaluation, offering similar capabilities for assessing retrieval quality and model performance.

For RAG evaluation, RAGAS is widely used to assess retrieval-augmented models, while ARES focuses on measuring how well the retrieved context supports the generated response.

Opik stands out as an open-source solution that provides structured evaluations, benchmarking, and observability for LLM applications, ensuring assessment transparency and consistency.

Other proprietary alternatives include Langfuse, Langsmith, which is deeply integrated into the LangChain ecosystem for debugging and evaluation, and Phoenix.

6. Architecting the observability pipeline

In our observability pipeline, implemented with Opik, we combine monitoring and evaluation to ensure our application runs smoothly. Monitoring tracks all activities, while evaluation assesses performance and correctness.

What’s the interface of the pipeline?

LLMOps observability pipelines consist of two parts: one for monitoring prompts and another for evaluating the RAG module. These pipelines help us track system performance and ensure the application remains reliable.

The prompt monitoring pipeline captures entire prompt traces and metadata, such as prompt templates or models used within the chain. It also logs latency and system behavior while providing structured insights through dashboards that help detect and resolve inefficiencies.

The RAG evaluation pipeline tests the agentic RAG module using heuristics and LLM judges to assess performance. It receives a set of evaluation prompts and processes them to evaluate accuracy and reasoning quality. The pipeline outputs accuracy assessments, quality scores, and alerts for performance issues, helping maintain system reliability.

We utilize Opik (by Comet ML), an open-source platform, to handle both the monitoring and evaluation of our application. Opik offers comprehensive tracing, automated evaluations, and production-ready dashboards, making it an ideal choice for our needs.

For evaluation, Opik automates performance assessments using both built-in and custom metrics. Users can define a threshold for any metric and configure alerts for immediate intervention if performance falls below the set value.

Now that we have an overview of the interfaces and components let’s dive into more details about each of the 2 pipelines.

The prompt monitoring pipeline

This component logs and monitors prompt traces. Prompt monitoring is essential to understand how our application interacts with users and identify areas for improvement. By tracking prompts and responses, we can debug issues in LLM reasoning or other issues like latency and costs.

Opik enables us to monitor latency across every phase of the generation process—pre-generation, generation, and post-generation—ensuring our application responds promptly to user inputs.

Latency is crucial to the user experience, as it includes multiple factors such as Time to First Token (TTFT), Time Between Tokens (TBT), Tokens Per Second (TPS), and Total Latency. Tracking these metrics helps us optimize response generation and manage hosting costs effectively.

Figure 5 provides an overview of how Opik logs and tracks prompt traces, helping us analyze inputs, outputs, and execution times for better performance monitoring.

Figure 5: Opik dashboard displaying the logged prompt traces

You can visualize details of the execution flow of a prompt, including its input, output, and latency at each step, as displayed in Figure 6. It helps us track the steps taken during processing, analyze latency at each stage, and identify potential inefficiencies.

Figure 6: Details of a prompt trace stored by Opik

Finally, in Figure 7, we can also visualize key metadata like retrieval parameters, system prompts, and model settings, providing deeper insights into prompt execution context:

Figure 7: The metadata of a prompt trace in Opik

For more details on observability with Opik, check out their documentation:

MORE ON OBSERVABILITY WITH OPIK

The last step is to understand the RAG evaluation pipeline.

The RAG evaluation pipeline

As previously mentioned, the RAG evaluation pipeline assesses the performance of our agentic RAG module, which performs application/RAG evaluation.

The pipeline uses built-in heuristics such as Hallucination, Answer Relevance, and Moderation to evaluate response quality. Additionally, we define and integrate a custom metric and LLM judge, which assesses if the LLM's output has appropriate length and density.

This flow can also run as an offline batch pipeline during development to assess performance on test sets. Additionally, it integrates into the CI/CD pipeline to test the RAG application before deployment, ensuring any issues are identified early (similar to integration tests).

Post-deployment, it can run on a schedule to evaluate random samples from production, maintaining consistent application performance. If metrics fall below a certain threshold, we can hook an alarm system that notifies us to address potential issues promptly.

Figure 8 illustrates the results of an evaluation experiment conducted on our RAG module. It displays both the built-in and the custom performance metrics configured by us.

Figure 8: Results of a RAG evaluation experiment stored in Opik

Opik allows us to compare multiple experiments side by side. This comparison helps track performance trends over time, making refining and improving our models easier.

Figure 9: Comparing 2 experiments in Opik

By implementing these components with Opik, we maintain a robust observability pipeline that ensures our application operates efficiently.

A final note is how similar a prompt management tool, such as Opik, is to more standard experiment tracking tools, such as Comet, W&B and MLFlow. But instead of being focused on simple metrics, it’s built around the prompts as their first-class citizen.

For more details on evaluation with Opik, check out their documentation:

MORE ON EVALUATION WITH OPIK

Finally, let’s dig into the implementation.

7. Implementing the agentic RAG module

Now that we’ve understood what it takes to build the agentic RAG and observability pipelines, let’s start implementing them.

The agentic RAG module is implemented using the SmolAgents Hugging Face frame, to build an agent that utilizes three key tools: the MongoDB retriever, the summarizer, and the "What Can I Do" tool.

Since prompt monitoring is closely tied to agent execution, here we will also cover how the system logs input/output data, latency, and other key details for each tool, ensuring full observability with Opik.

Building the agent

The core of our agentic RAG module starts with get_agent(), a method responsible for initializing the agent:

def get_agent(retriever_config_path: Path) -> "AgentWrapper":
    agent = AgentWrapper.build_from_smolagents(
        retriever_config_path=retriever_config_path
    )
    return agent

This function builds an AgentWrapper, which is a custom class we implemented that extends the agent's functionality by incorporating Opik for tracking all the agent’s interactions.

Building the agent requires a retriever configuration to create the MongoDB retriever tool. As a reminder from Lesson 5, we support multiple retrieval strategies based on retriever type (e.g., parent or contextual), embedding models, and other parameters.

Note: The retrieval setup is essentially copied from the offline Second Brain app in Lesson 5, ensuring consistency in document search and retrieval methods. This means the retriever is loaded exactly as it was implemented in the previous version, preserving the same retrieval logic and configurations.

Wrapping the agent for monitoring

The AgentWrapper class extends the base agent to incorporate metadata tracking with Opik. This ensures that every action taken by the agent is logged and traceable:

class AgentWrapper:
    def __init__(self, agent: MultiStepAgent) -> None:
        self.__agent = agent

    @property
    def input_messages(self) -> list[dict]:
        return self.__agent.input_messages

    @property
    def agent_name(self) -> str:
        return self.__agent.agent_name

    @property
    def max_steps(self) -> str:
        return self.__agent.max_steps

We use composition to wrap the MultiStepAgent from SmolAgents and expose its properties. The MultiStepAgent enables our agent to execute multi-step reasoning and decision-making processes.

Next, we define a method to build the agent, specifying the retriever configuration and integrating the 3 tools necessary for execution:

@classmethod
    def build_from_smolagents(cls, retriever_config_path: Path) -> "AgentWrapper":
        retriever_tool = MongoDBRetrieverTool(config_path=retriever_config_path)
        if settings.USE_HUGGINGFACE_DEDICATED_ENDPOINT:
            logger.warning(
                f"Using Hugging Face dedicated endpoint as the summarizer with URL: {settings.HUGGINGFACE_DEDICATED_ENDPOINT}"
            )
            summarizer_tool = HuggingFaceEndpointSummarizerTool()
        else:
            logger.warning(
                f"Using OpenAI as the summarizer with model: {settings.OPENAI_MODEL_ID}"
            )
            summarizer_tool = OpenAISummarizerTool(stream=False)

        model = LiteLLMModel(
            model_id=settings.OPENAI_MODEL_ID,
            api_base="https://api.openai.com/v1",
            api_key=settings.OPENAI_API_KEY,
        )

        agent = ToolCallingAgent(
            tools=[what_can_i_do, retriever_tool, summarizer_tool],
            model=model,
            max_steps=3,
            verbosity_level=2,
        )

        return cls(agent)

This method builds the agent by selecting the retriever configuration, which defines how the MongoDB retriever tool is created and configured.

It’s critical that the retriever config matches the one used during the RAG feature pipeline used to populate the MongoDB vector index.

Next, we build the summarizer tool, which can either be the custom model trained in Lesson 4 and deployed on Hugging Face or an OpenAI model, depending on the settings.

After that, we initialize the LiteLLM model, which powers our AI agent.

Finally, all tools, along with the LLM model, are wrapped inside a ToolCallingAgent class with a maximum of three reasoning steps, ensuring structured decision-making and controlled execution flow.

Now that our agent is built, we can define its run function:

@opik.track(name="Agent.run")
    def run(self, task: str, **kwargs) -> Any:
        result = self.__agent.run(task, **kwargs)

        model = self.__agent.model
        metadata = {
            "system_prompt": self.__agent.system_prompt,
            "system_prompt_template": self.__agent.system_prompt_template,
            "tool_description_template": self.__agent.tool_description_template,
            "tools": self.__agent.tools,
            "model_id": self.__agent.model.model_id,
            "api_base": self.__agent.model.api_base,
            "input_token_count": model.last_input_token_count,
            "output_token_count": model.last_output_token_count,
        }
        if hasattr(self.__agent, "step_number"):
            metadata["step_number"] = self.__agent.step_number
        opik_context.update_current_trace(
            tags=["agent"],
            metadata=metadata,
        )

        return result

The run method tracks every execution of the agent using Opik’s @track() decorator. It logs key metadata, including the system prompt, tool descriptions, model details, and token counts within the current trace.

Having the skeleton of our agent in place, we can dig into each of the 3 tools that our model calls.

Building the MongoDB retriever tool

The first tool integrated is the MongoDBRetrieverTool, which allows the agent to find relevant documents using semantic search.

It matches a user query to the most relevant stored documents, helping the agent retrieve context when needed.

To integrate the tool with our agent, we must inherit from the Tool class from SmolAgents. We also have to specify the name, description, inputs, and output type that the LLM uses to infer what the tool does and what its interface is. These are critical elements in integrating your tool with an LLM, as they are the only properties used to integrate the tool with the LLM:

class MongoDBRetrieverTool(Tool):
    name = "mongodb_vector_search_retriever"
    description = """Use this tool to search and retrieve relevant documents from a knowledge base using semantic search.
    This tool performs similarity-based search to find the most relevant documents matching the query.
    Best used when you need to:
    - Find specific information from stored documents
    - Get context about a topic
    - Research historical data or documentation
    The tool will return multiple relevant document snippets."""

    inputs = {
        "query": {
            "type": "string",
            "description": """The search query to find relevant documents for using semantic search.
            Should be a clear, specific question or statement about the information you're looking for.""",
        }
    }
    output_type = "string"

    def __init__(self, config_path: Path, **kwargs):
        super().__init__(**kwargs)

        self.config_path = config_path
        self.retriever = self.__load_retriever(config_path)

    def __load_retriever(self, config_path: Path):
        config = yaml.safe_load(config_path.read_text())
        config = config["parameters"]

        return get_retriever(
            embedding_model_id=config["embedding_model_id"],
            embedding_model_type=config["embedding_model_type"],
            retriever_type=config["retriever_type"],
            k=5,
            device=config["device"],
        )

The retriever tool is initialized with parameters from one of the retriever config files defined in Lesson 5. The settings include essential parameters such as the embedding model and retrieval type.

Now, we get to the core part of the tool, which is the forward method. This method is called when the AI agent uses the tool to search for information.

The forward method takes a query from the agent, searches for relevant documents, and returns the results in a format the agent can use.

The method is decorated with @track, which means its performance is being monitored with Opik. Before performing the actual search, the method first extracts important search parameters:

@track(name="MongoDBRetrieverTool.forward")
    def forward(self, query: str) -> str:
        if hasattr(self.retriever, "search_kwargs"):
            search_kwargs = self.retriever.search_kwargs
        else:
            try:
                search_kwargs = {
                    "fulltext_penalty": self.retriever.fulltext_penalty,
                    "vector_score_penalty": self.retriever.vector_penalty,
                    "top_k": self.retriever.top_k,
                }
            except AttributeError:
                logger.warning("Could not extract search kwargs from retriever.")

                search_kwargs = {}

        opik_context.update_current_trace(
            tags=["agent"],
            metadata={
                "search": search_kwargs,
                "embedding_model_id": self.retriever.vectorstore.embeddings.model,
            },
        )

First, we check what type of retriever is used and extract the relevant search parameters. Different retrievers might have different ways of configuring searches, so this code handles various cases.

The key parameters being extracted include:

fulltext_penalty: Adjusts how much weight is given to exact text matches
vector_score_penalty: Influences how semantic similarity affects the ranking
top_k: Determines how many search results to return

These parameters significantly impact the search results. For example, a higher vector score penalty might prioritize results that match the semantic meaning of the query over those with exact keyword matches.

After setting up tracking, the method parses the query, performs the actual search, and formats the results:

 try:
            query = self.__parse_query(query)
            relevant_docs = self.retriever.invoke(query)

            formatted_docs = []
            for i, doc in enumerate(relevant_docs, 1):
                formatted_docs.append(
                    f"""
<document id="{i}">
<title>{doc.metadata.get("title")}</title>
<url>{doc.metadata.get("url")}</url>
<content>{doc.page_content.strip()}</content>
</document>
"""
                )

            result = "\n".join(formatted_docs)
            result = f"""
<search_results>
{result}
</search_results>
When using context from any document, also include the document URL as reference, which is found in the <url> tag.
"""
            return result
        except Exception:
            logger.opt(exception=True).debug("Error retrieving documents.")

            return "Error retrieving documents."

In this code snippet, we search for documents that match the query and format them in an XML-like structure. Each document includes a title, URL, and content. Additionally, the results are wrapped in tags to make them easy for the AI agent to read.

Creating the summarizer tool

In our agentic RAG module, we provide two summarization options: one using Hugging Face’s API and another using OpenAI’s models. Both tools inherit from Tool in SmolAgents and are tracked by Opik, ensuring that every summarization step is logged and monitored.

The first option for summarization is the Hugging Face endpoint-based summarizer.

This tool sends the text to an external Hugging Face model that generates a concise summary. The model deployed on Hugging Face is the one we trained in Lesson 4, which was explicitly fine-tuned for document summarization.

class HuggingFaceEndpointSummarizerTool(Tool):
    name = "huggingface_summarizer"
    description = """Use this tool to summarize a piece of text. Especially useful when you need to summarize a document."""

    inputs = {
        "text": {
            "type": "string",
            "description": """The text to summarize.""",
        }
    }
    output_type = "string"

    SYSTEM_PROMPT = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
You are a helpful assistant specialized in summarizing documents. Generate a concise TL;DR summary in markdown format having a maximum of 512 characters of the key findings from the provided documents, highlighting the most significant insights

### Input:
{content}

### Response:
"""

    def __init__(self, *args, **kwargs) -> None:
        super().__init__(*args, **kwargs)

        assert settings.HUGGINGFACE_ACCESS_TOKEN is not None, (
            "HUGGINGFACE_ACCESS_TOKEN is required to use the dedicated endpoint. Add it to the .env file."
        )
        assert settings.HUGGINGFACE_DEDICATED_ENDPOINT is not None, (
            "HUGGINGFACE_DEDICATED_ENDPOINT is required to use the dedicated endpoint. Add it to the .env file."
        )

        self.__client = OpenAI(
            base_url=settings.HUGGINGFACE_DEDICATED_ENDPOINT,
            api_key=settings.HUGGINGFACE_ACCESS_TOKEN,
        )

The code snippet above initializes the Hugging Face summarizer tool. It verifies that the necessary API credentials are available before setting up the client connection to Hugging Face’s inference endpoint.

To generate a summary, we implement the forward method, which is tracked by Opik for monitoring:

@track
    def forward(self, text: str) -> str:
        result = self.__client.chat.completions.create(
            model="tgi",
            messages=[
                {
                    "role": "user",
                    "content": self.SYSTEM_PROMPT.format(content=text),
                },
            ],
        )

        return result.choices[0].message.content

This function sends the input text to the Hugging Face API, applying the predefined system prompt. The generated response is then returned, providing a structured summary.

The second summarization option uses OpenAI’s models to generate summaries. It follows a similar structure to the Hugging Face summarizer but connects to OpenAI’s API instead.

class OpenAISummarizerTool(Tool):
    name = "openai_summarizer"
    description = """Use this tool to summarize a piece of text. Especially useful when you need to summarize a document or a list of documents."""

    inputs = {
        "text": {
            "type": "string",
            "description": """The text to summarize.""",
        }
    }
    output_type = "string"

    SYSTEM_PROMPT = """You are a helpful assistant specialized in summarizing documents.
Your task is to create a clear, concise TL;DR summary in plain text.
Things to keep in mind while summarizing:
- titles of sections and sub-sections
- tags such as Generative AI, LLMs, etc.
- entities such as persons, organizations, processes, people, etc.
- the style such as the type, sentiment and writing style of the document
- the main findings and insights while preserving key information and main ideas
- ignore any irrelevant information such as cookie policies, privacy policies, HTTP errors,etc.

Document content:
{content}

Generate a concise summary of the key findings from the provided documents, highlighting the most significant insights and implications.
Return the document in plain text format regardless of the original format.
"""

    def __init__(self, *args, **kwargs) -> None:
        super().__init__(*args, **kwargs)

        self.__client = OpenAI(
            base_url="https://api.openai.com/v1",
            api_key=settings.OPENAI_API_KEY,
        )

This summarizer connects to OpenAI’s API and uses a structured prompt to generate high-quality summaries.

Note that because the Hugging Face model was fine-tuned on summarizing documents, it doesn't require careful prompt engineering for the desired results (it has the logic embedded into it), resulting in fewer tokens/requests, which translates to lower costs and better latencies.

The "What Can I Do" tool

The third and last integrated tool is the "What Can I Do" tool, which provides a list of available capabilities within the Second Brain assistant and helps users explore relevant topics.

@opik.track(name="what_can_i_do")
@tool
def what_can_i_do(question: str) -> str:
    """Returns a comprehensive list of available capabilities and topics in the Second Brain system.

    This tool should be used when:
    - The user explicitly asks what the system can do
    - The user asks about available features or capabilities
    - The user seems unsure about what questions they can ask
    - The user wants to explore the system's knowledge areas

    This tool should NOT be used when:
    - The user asks a specific technical question
    - The user already knows what they want to learn about
    - The question is about a specific topic covered in the knowledge base

    Args:
        question: The user's query about system capabilities. While this parameter is required,
                 the function returns a standard capability list regardless of the specific question.

    Returns:
        str: A formatted string containing categorized lists of example questions and topics
             that users can explore within the Second Brain system.

    Examples:
        >>> what_can_i_do("What can this system do?")
        >>> what_can_i_do("What kind of questions can I ask?")
        >>> what_can_i_do("Help me understand what I can learn here")
    """

    return """
You can ask questions about the content in your Second Brain, such as:

Architecture and Systems:
- What is the feature/training/inference (FTI) architecture?
- How do agentic systems work?
- Detail how does agent memory work in agentic applications?

LLM Technology:
- What are LLMs?
- What is BERT (Bidirectional Encoder Representations from Transformers)?
- Detail how does RLHF (Reinforcement Learning from Human Feedback) work?
- What are the top LLM frameworks for building applications?
- Write me a paragraph on how can I optimize LLMs during inference?

RAG and Document Processing:
- What tools are available for processing PDFs for LLMs and RAG?
- What's the difference between vector databases and vector indices?
- How does document chunk overlap affect RAG performance?
- What is chunk reranking and why is it important?
- What are advanced RAG techniques for optimization?
- How can RAG pipelines be evaluated?

Learning Resources:
- Can you recommend courses on LLMs and RAG?
"""

This tool is useful when users are unsure about what they can ask or want to explore different capabilities within the system. Like other tools, it is tracked by Opik for monitoring and observability.

To see our agentic RAG module in action, check out the video below, where we query our agent using the Gradio UI, visualizing how the agent reasons and calls the tools to construct the answer to our question:

Having the agentic module tested, we can check out the results of the tracking done by Opik in Figure 10:

Figure 10: Example of a prompt trace in Opik

Here, we can see that the agent calls the MongoDB retriever tool, which in turn invokes the forward function. Each step is logged with latency values, providing insight into execution times at different stages.

Furthermore, all metadata related to the trace—including the system prompt, tool configurations, and token usage—is captured to ensure complete observability.

8. Building the LLM evaluation pipeline

Now that we have implemented the agentic RAG module, we need a structured way to evaluate its performance. This is where the LLM evaluation pipeline comes in, ensuring that our agentic RAG module consistently meets quality and reliability standards.

The evaluation pipeline is built using Opik, which helps us log, analyze, and score the agent’s responses. We will focus strictly on Opik's evaluation logic and how it tracks our agent’s outputs.

Before evaluating our agent, we first need to gather a suitable evaluation dataset. This dataset will help us consistently test performance and track improvements

Creating the evaluation dataset

To evaluate the agent properly, we use a dataset of ~30 predefined prompts that cover various scenarios the agent might encounter. This dataset allows us to consistently test our agent’s performance across different iterations, ensuring that changes do not degrade its capabilities.

EVALUATION_PROMPTS: List[str] = [
    """
Write me a paragraph on the feature/training/inference (FTI) pipelines architecture following the next structure:

- introduction
- what are its main components
- why it's powerful

Retrieve the sources when compiling the answer. Also, return the sources you used as context.
""",
    "What is the feature/training/inference (FTI) pipelines architecture?",
    "What is the Tensorflow Recommenders Python package?",
    """How does RLHF: Reinforcement Learning from Human Feedback work?

Explain to me:
- what is RLHF
- how it works
- why it's important
- what are the main components
- what are the main challenges
""",
    "List 3 LLM frameworks for building LLM applications and why they are important.",
    "Explain how does Bidirectional Encoder Representations from Transformers (BERT) work. Focus on what architecture it uses, how it's different from other models and how they are trained.",
    "List 5 ways or tools to process PDFs for LLMs and RAG",
    """How can I optimize my LLMs during inference?

Provide a list of top 3 best practices, while providing a short explanation for each, which contains why it's important.
""",
    "Explain to me in more detail how does an Agent memory work and why do we need it when building Agentic apps.",
    "What is the difference between a vector database and a vector index?",
    "Recommend me a course on LLMs and RAG",
    "How Document Chunk overlap affects a RAG pipeline and it's performance?",
    """What is the importance of reranking chunks for RAG?
Explain to me:
- what is reranking
- how it works
- why it's important
- what are the main components
- what are the main trade-offs
""",
    "List the most popular advanced RAG techniques to optimize RAG performance and why they are important.",
    "List what are the main ways of evaluating a RAG pipeline and why they are important.",
]

We could have added more samples, but for the first iteration, having 30 samples is a sweet spot. The core idea is to expand this split with edge case samples you find while developing the application.

We use Opik to store and manage the dataset, as shown in the following code:

def get_or_create_dataset(name: str, prompts: list[str]) -> opik.Dataset | None:
    client = opik.Opik()
    try:
        dataset = client.get_dataset(name=name)
    except Exception:
        dataset = None

    if dataset:
        logger.warning(f"Dataset '{name}' already exists. Skipping dataset creation.")

        return dataset

    assert prompts, "Prompts are required to create a dataset."

    dataset_items = []
    for prompt in prompts:
        dataset_items.append(
            {
                "input": prompt,
            }
        )

    dataset = create_dataset(
        name=name,
        description="Dataset for evaluating the agentic app.",
        items=dataset_items,
    )

    return dataset

This function ensures the dataset is created if it doesn’t exist, avoiding unnecessary duplication. It logs whether the dataset is new or previously stored and ensures that each prompt is properly formatted before evaluation.

Evaluating the agent

The core of the evaluation pipeline is the evaluate_agent() function. This function runs the set of predefined prompts through our agent and scores its responses using a combination of built-in and custom metrics.

def evaluate_agent(prompts: list[str], retriever_config_path: Path) -> None:
    assert settings.COMET_API_KEY, (
        "COMET_API_KEY is not set. We need it to track the experiment with Opik."
    )

    logger.info("Starting evaluation...")
    logger.info(f"Evaluating agent with {len(prompts)} prompts.")

    def evaluation_task(x: dict) -> dict:
        """Call agentic app logic to evaluate."""
        agent = agents.get_agent(retriever_config_path=retriever_config_path)
        response = agent.run(x["input"])
        context = extract_tool_responses(agent)

        return {
            "input": x["input"],
            "context": context,
            "output": response,
        }

In this code section, we first ensure that Opik can log the experiment by asserting that the necessary API keys are set.

Then, we define the evaluation_task(), a method that retrieves an instance of our agent, runs an input prompt through it, and captures both the output and retrieval context.

Before running the actual evaluation, we either fetch an existing dataset or create a new one to store our evaluation prompts:

# Get or create dataset
    dataset_name = "second_brain_rag_agentic_app_evaluation_dataset"
    dataset = opik_utils.get_or_create_dataset(name=dataset_name, prompts=prompts)

Here, opik_utils.get_or_create_dataset() is used to manage the datasets dynamically, as detailed earlier in this section.

Once the dataset is set up, we retrieve our agent instance and configure the experiment. The experiment_config dictionary defines key parameters for tracking and logging the evaluation:

# Evaluate
    agent = agents.get_agent(retriever_config_path=retriever_config_path)
    experiment_config = {
        "model_id": settings.OPENAI_MODEL_ID,
        "retriever_config_path": retriever_config_path,
        "agent_config": {
            "max_steps": agent.max_steps,
            "agent_name": agent.agent_name,
        },
    }

Next, we define the scoring metrics used to evaluate the agent's performance. Opik provides built-in evaluation metrics, but we also include custom ones for deeper analysis.

scoring_metrics = [
        Hallucination(),
        AnswerRelevance(),
        Moderation(),
        SummaryDensityHeuristic(),
        SummaryDensityJudge(),
    ]

The scoring process evaluates the agent’s performance across multiple dimensions:

Hallucination: Measures whether the agent generates false or misleading information.
Answer Relevance: Scores the relevance of the agent's response to the given prompt.
Moderation: Detects potentially inappropriate or unsafe content in responses.

For more details on the metrics above or on how to build custom metrics, check out Opik’s docs.

In addition to these built-in Opik metrics, we include two custom components. Both compute the response density (whether the answer is too long or too short) but with different techniques: heuristics or LLM-as-Judges. This is a good example of understanding the difference between the two.

SummaryDensityHeuristic: Evaluates whether a response is too short, too long, or appropriately balanced.
SummaryDensityJudge: Uses an external LLM to judge response density and conciseness.

Finally, we execute the evaluation process using the metrics defined and our evaluation dataset:

if dataset:
        evaluate(
            dataset=dataset,
            task=evaluation_task,
            scoring_metrics=scoring_metrics,
            experiment_config=experiment_config,
            task_threads=2,
        )
    else:
        logger.error("Can't run the evaluation as the dataset items are empty.")

This code ensures that evaluation runs only when a dataset is available. The evaluate() function runs the agent using the evaluation_task() method on the evaluation dataset and measures the defined scoring metrics. The results are then logged in Opik for further analysis and comparison.

MORE ON OPIK'S METRICS

The summary density heuristic

In our evaluation pipeline, we include a custom metric called summary density heuristic.

This metric assesses whether an LLM-generated response is appropriately concise and informative. It extends BaseMetric from Opik, allowing us to integrate it seamlessly into our evaluation framework.

The purpose of this heuristic is to ensure that responses are neither too short nor excessively long. A well-balanced response provides sufficient detail without unnecessary verbosity.

class SummaryDensityHeuristic(base_metric.BaseMetric):
    """
    A metric that evaluates whether an LLM's output has appropriate length and density.

    This metric uses an heuristic to determine if the output length is appropriate for the given instruction.
    It returns a normalized score between 0 and 1, where:
    - 0.0 (Poor): Output is either too short and incomplete, or too long with unnecessary information
    - 0.5 (Good): Output has decent length balance but still slightly too short or too long
    - 1.0 (Excellent): Output length is appropriate, answering the question concisely without being verbose
    """

    def __init__(
        self,
        name: str = "summary_density_heuristic",
        min_length: int = 128,
        max_length: int = 1024,
    ) -> None:
        self.name = name
        self.min_length = min_length
        self.max_length = max_length

This snippet initializes the metric with a name, minimum length, and maximum length. The min_length and max_length parameters define the acceptable range for a response's length.

To evaluate the response length, we define the score() function, which compares the output against the predefined length limits:

 def score(
        self, input: str, output: str, **ignored_kwargs: Any
    ) -> score_result.ScoreResult:
        """
        Score the output of an LLM.

        Args:
            input: The input prompt given to the LLM.
            output: The output of an LLM to score.
            **ignored_kwargs: Any additional keyword arguments.

        Returns:
            ScoreResult: The computed score with explanation.
        """

        length_score = self._compute_length_score(output)

        reason = f"Output length: {len(output)} chars. "
        if length_score == 1.0:
            reason += "Length is within ideal range."
        elif length_score >= 0.5:
            reason += "Length is slightly outside ideal range."
        else:
            reason += "Length is significantly outside ideal range."

        return score_result.ScoreResult(
            name=self.name,
            value=length_score,
            reason=reason,
        )

The score() function determines how well the LLM's response fits within the acceptable length range. It assigns a normalized score between 0 and 1 based on whether the output is too short, too long, or appropriately balanced.

The core logic of this metric lies in _compute_length_score(), which calculates a numerical score based on response length:

    def _compute_length_score(self, text: str) -> float:
        """
        Compute a score based on text length relative to min and max boundaries.

        Args:
            text: The text to evaluate.

        Returns:
            float: A score between 0 and 1, where:
                - 0.0: Text length is significantly outside the boundaries
                - 0.5: Text length is slightly outside the boundaries
                - 1.0: Text length is within the ideal range
        """
        length = len(text)

        # If length is within bounds, return perfect score
        if self.min_length <= length <= self.max_length:
            return 1.0

        if length < self.min_length:
            deviation = (self.min_length - length) / self.min_length
        else:
            deviation = (length - self.max_length) / self.max_length

        # Convert deviation to a score between 0 and 1
        # deviation <= 0.5 -> score between 0.5 and 1.0
        # deviation > 0.5 -> score between 0.0 and 0.5
        score = max(0.0, 1.0 - deviation)

        return score

This function ensures that responses falling within the predefined length range receive a perfect score of 1.0. If a response deviates too far from the range, its score is gradually reduced to reflect the severity of the deviation.

The summary density judge

The summary density judge is a custom evaluation component that builds upon the summary density metric by using an external LLM to assess response length.

Instead of relying on a manually calculated heuristic, this judge uses an AI model to determine if the length of an output is appropriate for a given input.

This approach allows us to incorporate more nuanced and context-aware judgments into our evaluation pipeline. Like the heuristic, it integrates seamlessly with Opik’s evaluation framework:

class LLMJudgeStyleOutputResult(BaseModel):
    score: int
    reason: str


class SummaryDensityJudge(base_metric.BaseMetric):
    """
    A metric that evaluates whether an LLM's output has appropriate length and density.

    This metric uses another LLM to judge if the output length is appropriate for the given instruction.
    It returns a normalized score between 0 and 1, where:
    - 0.0 (Poor): Output is either too short and incomplete, or too long with unnecessary information
    - 0.5 (Good): Output has decent length balance but still slightly too short or too long
    - 1.0 (Excellent): Output length is appropriate, answering the question concisely without being verbose
    """

    def __init__(
        self,
        name: str = "summary_density_judge",
        model_name: str = settings.OPENAI_MODEL_ID,
    ) -> None:
        self.name = name
        self.llm_client = LiteLLMChatModel(model_name=model_name)
        self.prompt_template = """
        You are an impartial expert judge. Evaluate the quality of a given answer to an instruction based on how long the answer it is. 

How to decide wether the lengths of the answer is appropriate:
1 (Poor): Too short, does not answer the question OR too long, it contains too much noise and unrequired information, where the answer could be more concise.
2 (Good): Good lengthbalance of the answer, but the answer is still too short OR too long.
3 (Excellent): The length of the answer is appropriate, it answers the question and is not too long or too short.

Example of bad answer that is too short: 
<answer>
LangChain, LlamaIndex, Haystack
</answer>

Example of bad answer that is too long:
<answer>
LangChain is a powerful and versatile framework designed specifically for building sophisticated LLM applications. It provides comprehensive abstractions for essential components like prompting, memory management, agent behaviors, and chain orchestration. The framework boasts an impressive ecosystem with extensive integrations across various tools and services, making it highly flexible for diverse use cases. However, this extensive functionality comes with a steeper learning curve that might require dedicated time to master.

LlamaIndex (which was formerly known as GPTIndex) has carved out a specialized niche in the LLM tooling landscape, focusing primarily on data ingestion and advanced indexing capabilities for Large Language Models. It offers a rich set of sophisticated mechanisms to structure and query your data, including vector stores for semantic similarity search, keyword indices for traditional text matching, and tree indices for hierarchical data organization. While it particularly shines in Retrieval-Augmented Generation (RAG) applications, its comprehensive feature set might be excessive for more straightforward implementation needs.

Haystack stands out as a robust end-to-end framework that places particular emphasis on question-answering systems and semantic search capabilities. It provides a comprehensive suite of document processing tools and comes equipped with production-ready pipelines that can be deployed with minimal configuration. The framework includes advanced features like multi-stage retrieval, document ranking, and reader-ranker architectures. While these capabilities make it powerful for complex information retrieval tasks, new users might find the initial configuration and architecture decisions somewhat challenging to navigate.

Each of these frameworks brings unique strengths to the table while sharing some overlapping functionality. The choice between them often depends on specific use cases, technical requirements, and team expertise. LangChain offers the broadest general-purpose toolkit, LlamaIndex excels in data handling and RAG, while Haystack provides the most streamlined experience for question-answering systems.
</answer>

Example of excellent answer that is appropriate:
<answer>
1. LangChain is a powerful framework for building LLM applications that provides abstractions for prompting, memory, agents, and chains. It has extensive integrations with various tools and services, making it highly flexible but potentially complex to learn. 
2. LlamaIndex specializes in data ingestion and indexing for LLMs, offering sophisticated ways to structure and query your data through vector stores, keyword indices, and tree indices. It excels at RAG applications but may be overkill for simpler use cases. 
3. Haystack is an end-to-end framework focused on question-answering and semantic search, with strong document processing capabilities and ready-to-use pipelines. While powerful, its learning curve can be steep for beginners. 
</answer>

Instruction: {input}

Answer: {output}

Provide your evaluation in JSON format with the following structure:
{{
    "accuracy": {{
        "reason": "...",
        "score": 0
    }},
    "style": {{
        "reason": "...",
        "score": 0
    }}
}}
"""

In this snippet, we initialize the summary density judge, specifying the model it will use to evaluate responses. The prompt_template provides clear instructions for the external LLM, defining the criteria for scoring an answer.

The judge’s scoring function uses the external LLM to analyze a response and assign a score based on how well its length aligns with the expected range:

    def score(self, input: str, output: str, **ignored_kwargs: Any):
        prompt = self.prompt_template.format(input=input, output=output)

        model_output = self.llm_client.generate_string(
            input=prompt, response_format=LLMJudgeStyleOutputResult
        )

        return self._parse_model_output(model_output)

The score() function formats the prompt and sends it to the external LLM. The model then evaluates the response and provides a structured output with a score and explanation.

Once the external model returns a score, we process it to ensure consistency and normalize the values.

    def _parse_model_output(self, content: str) -> score_result.ScoreResult:
        try:
            dict_content = json.loads(content)
        except Exception:
            raise exceptions.MetricComputationError("Failed to parse the model output.")

        score = dict_content["score"]
        try:
            assert 1 <= score <= 3, f"Invalid score value: {score}"
        except AssertionError as e:
            raise exceptions.MetricComputationError(str(e))

        score = (score - 1) / 2.0  # Normalize the score to be between 0 and 1

        return score_result.ScoreResult(
            name=self.name,
            value=score,
            reason=dict_content["reason"],
        )

The _parse_model_output() function ensures that the returned score is valid and within the expected range. The score is then normalized between 0 and 1 for consistency with other evaluation metrics.

Evaluation results

We track evaluation results in Opik, allowing us to compare different agent versions and detect performance regressions.

Figure 11 shows a sample evaluation run, displaying the scores across all metrics:

Figure 11: Outcome of an evaluation run in Opik

By implementing this evaluation pipeline, we ensure that our agentic RAG module continues to improve while maintaining accuracy, relevance, and overall quality.

9. Running the code

The best way to set up and run the code is through our GitHub repository, where we have documented everything you need. We will keep the end-to-end instructions only in our GitHub to avoid having the documentation scattered throughout too many places (which is a pain to maintain and use).

First, you have to ensure that your MongoDB Docker container is running and that your RAG collection is populated.

Next, you can run the agent through the command-line interface (CLI) for a quick test or with a Gradio UI for a more interactive experience.

To quickly test the Agentic RAG inference on a predefined query, you can run the following command from the CLI:

make run_agent_query RETRIEVER_CONFIG=configs/compute_rag_vector_index_openai_parent.yaml

Note: The retriever config can be any of the ones defined in Lesson 5, depending on the retrieval strategy you want to use (but they have to match, between the RAG feature pipeline and the inference pipeline).

For a more interactive experience, you can launch the Gradio UI by executing:

make run_agent_app RETRIEVER_CONFIG=configs/compute_rag_vector_index_openai_parent.yaml

Additionally, if you want to evaluate the agent’s performance, run the evaluation pipeline using:

make evaluate_agent RETRIEVER_CONFIG=configs/compute_rag_vector_index_openai_parent.yaml

All the runs, including inference and evaluation, can be tracked directly from the Opik dashboards, providing insights into performance and enabling better monitoring of experiments.

For the whole setup and running guide, go to our GitHub:

GO TO GITHUB

Conclusion

This was a long lesson—if you're still here, you’ve made it to the end of the Building Your Second Brain AI Assistant course. Congrats!

Throughout this lesson, we explored LLM-powered agents and how they differ from traditional workflows.

We designed and implemented the Agentic RAG module, integrating SmolAgents, Gradio, and MongoDB to enable dynamic retrieval and reasoning. We then built an observability pipeline using Opik, ensuring full monitoring and evaluation of our agentic system.

Beyond implementation, we focused on evaluating and improving the agent's performance. We explored prompt monitoring, latency tracking, and response evaluation using built-in and custom metrics, including heuristic-based scoring and LLM-as-a-judge techniques.

With this final lesson, you now have a complete, end-to-end understanding of architecting, building, and evaluating LLM-powered AI assistants.

If you haven’t read all the lessons from the Second Brain AI Assistant open-source course, consider starting with Lesson 1 on architecting the end-to-end LLM system.

💻 Explore all the lessons and the code in our freely available GitHub repository.

If you have questions or need clarification, feel free to ask. See you in the next session!

Whenever you’re ready, here is how I can help you

Go from agent user to agent builder. Master the foundations of AI agents and turn fragile demo code into reliable, production-ready systems with my course, Agent Engineering: Building Multi-Agent Systems (made with Towards AI).

35 lessons. Pure foundations from scratch. 4 mini-projects. 2 production systems. A certificate and direct access to me & industry experts in our Discord.

Built for software and data professionals transitioning into AI engineering. Rated 5/5 with 300+ students. The first 7 lessons are free:

Start here

Not ready to commit? Start with our free Agent AI Engineering Guide, a 6-day email course on the mistakes that silently break AI agents in production.

References

Decodingml. (n.d.). GitHub - decodingml/second-brain-ai-assistant-course. GitHub. https://github.com/decodingml/second-brain-ai-assistant-course

Iusztin P., Labonne M. (2024, October 22). LLM Engineer’s Handbook | Data | Book. Packt. https://www.packtpub.com/en-us/product/llm-engineers-handbook-9781836200062

Kuligin, L., Zaldívar, J., & Tschochohei, M. (n.d.). Generative AI on Google Cloud with LangChain: Design scalable generative AI solutions with Python, LangChain, and Vertex AI on Google Cloud. https://www.amazon.com/Generative-Google-Cloud-LangChain-generative/dp/B0DKT8DCRT

Log traces. (n.d.). https://www.comet.com/docs/opik/tracing/log_traces

Varshney, T. (2024, June 24). Introduction to LLM Agents | NVIDIA Technical Blog. NVIDIA Technical Blog. https://developer.nvidia.com/blog/introduction-to-llm-agents/

What’s AI by Louis-François Bouchard. (2025, February 2). Real Agents vs. Workflows: The Truth Behind AI “Agents” [Video]. YouTube. https://www.youtube.com/watch?v=kQxr-uOxw2o

DecodingML. (n.d.). The Ultimate Prompt Monitoring Pipeline. Medium. https://medium.com/decodingml/the-ultimate-prompt-monitoring-pipeline-886cbb75ae25

DecodingML. (n.d.). The Engineer’s Framework for LLM RAG Evaluation. Medium. https://medium.com/decodingml/the-engineers-framework-for-llm-rag-evaluation-59897381c326

Cardenas, E., & Monigatti, L. (n.d.). What is Agentic RAG? Weaviate Blog. https://weaviate.io/blog/what-is-agentic-rag

Sponsors

Thank our sponsors for supporting our work — this course is free because of them!

Images

If not otherwise stated, all images are created by the author.