DML: Chain of Thought Reasoning: Write robust & explainable prompts for your LLM
Everything you need to know about chaining prompts: increase your LLMs accuracy & debug and explain your LLM.
Hello there, I am Paul Iusztin ๐๐ผ
Within this newsletter, I will help you decode complex topics about ML & MLOps one week at a time ๐ฅ
This weekโs ML & MLOps topics:
Chaining Prompts to Reduce Costs, Increase Accuracy & Easily Debug Your LLMs
Chain of Thought Reasoning: Write robust & explainable prompts for your LLM
Extra: Why any ML system should use an ML platform as its central nervous system
But first, I want to share with you this quick 7-minute guide teaching you how stable diffusion models are trained and generate new images.
Diffusion models are the cornerstone of most modern computer vision generative AI applications.
Thus, if you are into generative AI, it is essential to have an intuition of how a diffusion model works.
Check out my article to quickly understand:
- the general picture of how diffusion models work
- how diffusion models generate new images
- how they are trained
- how they are controlled by a given context (e.g., text)
โณ๐ Busy? This Is Your Quick Guide to Opening the Diffusion Models Black Box
#1. Chaining Prompts to Reduce Costs, Increase Accuracy & Easily Debug Your LLMs
Here it is โ
๐๐ต๐ฎ๐ถ๐ป๐ถ๐ป๐ด ๐ฝ๐ฟ๐ผ๐บ๐ฝ๐๐ is an intuitive technique that states that you must split your prompts into multiple calls.
๐ช๐ต๐? ๐๐ฒ๐'๐ ๐๐ป๐ฑ๐ฒ๐ฟ๐๐๐ฎ๐ป๐ฑ ๐๐ต๐ถ๐ ๐๐ถ๐๐ต ๐๐ผ๐บ๐ฒ ๐ฎ๐ป๐ฎ๐น๐ผ๐ด๐ถ๐ฒ๐.
When cooking, you are following a recipe split into multiple steps. You want to move to the next step only when you know what you have done so far is correct.
โณ You want every prompt to be simple & focused.
Another analogy is between reading all the code in one monolith/god class and using DRY to separate the logic between multiple modules.
โณ You want to understand & debug every prompt easily.
.
Chaining prompts is a ๐ฝ๐ผ๐๐ฒ๐ฟ๐ณ๐๐น ๐๐ผ๐ผ๐น ๐ณ๐ผ๐ฟ ๐ฏ๐๐ถ๐น๐ฑ๐ถ๐ป๐ด ๐ฎ ๐๐๐ฎ๐๐ฒ๐ณ๐๐น ๐๐๐๐๐ฒ๐บ where you must take different actions depending on the current state.
In other words, you control what happens between 2 chained prompts.
๐๐บ๐ฑ๐ณ๐ฐ๐ฅ๐ถ๐ค๐ต๐ด ๐ฐ๐ง ๐ค๐ฉ๐ข๐ช๐ฏ๐ช๐ฏ๐จ ๐ฑ๐ณ๐ฐ๐ฎ๐ฑ๐ต๐ด:
- increase in accuracy
- reduce the number of tokens -> lower costs (skips steps of the workflow when not needed)
- avoid context limitations
- easier to include a human-in-the-loop -> easier to control, moderate, test & debug
- use external tools/plugins (web search, API, databases, calculator, etc.)
.
๐๐
๐ฎ๐บ๐ฝ๐น๐ฒ
You want to build a virtual assistant to respond to customer service queries.
Instead of adding in one single prompt the system message, all the available products, and the user inquiry, you can split it into the following:
1. Use a prompt to extract the products and categories of interest.
2. Enrich the context only with the products of interest.
3. Call the LLM for the final answer.
You can evolve this example by adding another prompt that classifies the nature of the user inquiry. Based on that, redirect it to billing, technical support, account management, or a general LLM (similar to the complex system of GPT-4).
๐ง๐ผ ๐๐๐บ๐บ๐ฎ๐ฟ๐ถ๐๐ฒ:
Instead of writing a giant prompt that includes multiple steps:
Split the god prompt into multiple modular prompts that let you keep track of the state externally and orchestrate the program.
In other words, you want modular prompts that you can combine easily (same as in writing standard functions/classes)
.
To ๐ฎ๐๐ผ๐ถ๐ฑ ๐ผ๐๐ฒ๐ฟ๐ฒ๐ป๐ด๐ถ๐ป๐ฒ๐ฒ๐ฟ๐ถ๐ป๐ด, use this technique when your prompt contains >= instruction.
You can leverage the DRY principle from software -> one prompt = one instruction.
โณ๐ Tools to chain prompts: LangChain
โณ๐ Tools to monitor and debug prompts: Comet LLMOps Tools
#2. Chain of Thought Reasoning: Write robust & explainable prompts for your LLM
๐๐ต๐ฎ๐ถ๐ป ๐ผ๐ณ ๐ง๐ต๐ผ๐๐ด๐ต๐ ๐ฅ๐ฒ๐ฎ๐๐ผ๐ป๐ถ๐ป๐ด is a ๐ฝ๐ผ๐๐ฒ๐ฟ๐ณ๐๐น ๐ฝ๐ฟ๐ผ๐บ๐ฝ๐ ๐ฒ๐ป๐ด๐ถ๐ป๐ฒ๐ฒ๐ฟ๐ถ๐ป๐ด ๐๐ฒ๐ฐ๐ต๐ป๐ถ๐พ๐๐ฒ to ๐ถ๐บ๐ฝ๐ฟ๐ผ๐๐ฒ ๐๐ผ๐๐ฟ ๐๐๐ '๐ ๐ฎ๐ฐ๐ฐ๐๐ฟ๐ฎ๐ฐ๐ ๐ฎ๐ป๐ฑ ๐ฒ๐
๐ฝ๐น๐ฎ๐ถ๐ป ๐ถ๐๐ ๐ฎ๐ป๐๐๐ฒ๐ฟ.
Let me explain โ
It is a method to force the LLM to follow a set of predefined steps.
๐ง ๐ช๐ต๐ ๐ฑ๐ผ ๐๐ฒ ๐ป๐ฒ๐ฒ๐ฑ ๐๐ต๐ฎ๐ถ๐ป ๐ผ๐ณ ๐ง๐ต๐ผ๐๐ด๐ต๐ ๐ฅ๐ฒ๐ฎ๐๐ผ๐ป๐ถ๐ป๐ด?
In complex scenarios, the LLM must thoroughly reason about a problem before responding to the question.
Otherwise, the LLM might rush to an incorrect conclusion.
By forcing the model to follow a set of steps, we can guide the model to "think" more methodically about the problem.
Also, it helps us explain and debug how the model reached a specific answer.
.
๐ก ๐๐ป๐ป๐ฒ๐ฟ ๐ ๐ผ๐ป๐ผ๐น๐ผ๐ด๐๐ฒ
The inner monologue is all the steps needed to reach the final answer.
Often, we want to hide all the reasoning steps from the end user.
In fancy words, we want to mimic an "inner monologue" and output only the final answer.
Each reasoning step is structured into a parsable format.
Thus, we can quickly load it into a data structure and output only the desired steps to the user.
.
โณ ๐๐ฒ๐'๐ ๐ฏ๐ฒ๐๐๐ฒ๐ฟ ๐๐ป๐ฑ๐ฒ๐ฟ๐๐๐ฎ๐ป๐ฑ ๐๐ต๐ถ๐ ๐๐ถ๐๐ต ๐ฎ๐ป ๐ฒ๐
๐ฎ๐บ๐ฝ๐น๐ฒ:
The input prompt to the LLM consists of a system message + the user's question.
The secret is in defining the system message as follows:
"""
You are a virtual assistant helping clients...
Follow the next steps to answer the customer queries.
Step 1: Decide if it is a question about a product ...
Step 2: Retrieve the product ...
Step 3: Extract user assumptions ...
Step 4: Validate user assumptions ...
Step 5: Answer politely ...
Make sure to answer in the following format:
Step 1: <๐ด๐ต๐ฆ๐ฑ_1_๐ข๐ฏ๐ด๐ธ๐ฆ๐ณ>
Step 2: <๐ด๐ต๐ฆ๐ฑ_2_๐ข๐ฏ๐ด๐ธ๐ฆ๐ณ>
Step 3: <๐ด๐ต๐ฆ๐ฑ_3_๐ข๐ฏ๐ด๐ธ๐ฆ๐ณ>
Step 4: <๐ด๐ต๐ฆ๐ฑ_4_๐ข๐ฏ๐ด๐ธ๐ฆ๐ณ>
Response to the user: <๐ง๐ช๐ฏ๐ข๐ญ_๐ณ๐ฆ๐ด๐ฑ๐ฐ๐ฏ๐ด๐ฆ>
"""
Enforcing the LLM to follow a set of steps, we ensured it would answer the right questions.
Ultimately, we will show the user only the <๐ง๐ช๐ฏ๐ข๐ญ_๐ณ๐ฆ๐ด๐ฑ๐ฐ๐ฏ๐ด๐ฆ> subset of the answer.
The other steps (aka "inner monologue") help:
- the model to reason
- the developer to debug
Have you used this technique when writing prompts?
Extra: Why any ML system should use an ML platform as its central nervous system
Any ML system should use an ML platform as its central nervous system.
Here is why โ
The primary role of an ML Platform is to bring structure to your:
- experiments
- visualizations
- models
- datasets
- documentation
Also, its role is to decouple your data preprocessing, experiment, training, and inference pipelines.
.
An ML platform helps you automate everything mentioned above using these 6 features:
1. experiment tracking: log & compare experiments
2. metadata store: know how a model (aka experiment) was generated
3. visualisations: a central hub for your visualizations
4. reports: create documents out of your experiments
5. artifacts: version & share your datasets
6. model registry: version & share your models
I have used many ML Platforms before, but lately, I started using Comet, and I love it.
โณ๐ Comet ML
What is your favorite ML Platform?
Thatโs it for today ๐พ
See you next Thursday at 9:00 a.m. CET.
Have a fantastic weekend!
Paul
Whenever youโre ready, here is how I can help you:
The Full Stack 7-Steps MLOps Framework: a 7-lesson FREE course that will walk you step-by-step through how to design, implement, train, deploy, and monitor an ML batch system using MLOps good practices. It contains the source code + 2.5 hours of reading & video materials on Medium.
Machine Learning & MLOps Blog: in-depth topics about designing and productionizing ML systems using MLOps.
Machine Learning & MLOps Hub: a place where all my work is aggregated in one place (courses, articles, webinars, podcasts, etc.).