Alexey's leave-one-out trick for FAQ evals is a neat fix for the cold-start problem. The curation step that returns NEW, UPDATE, DUPLICATE or WRONG_COURSE is also a natural fit for a decision model rather than a chat model: fixed labels, a confidence score, and your code decides what happens next. I've been collecting guides and experiments for that pattern on JevMade.
This matches how I ended up working too. I build a RAG assistant for a content platform, and the thing that actually kept me honest was starting with deterministic checks before bringing in any LLM judge. Simple rules like "every claim needs a cited source" and "flag answers below a support threshold" catch most regressions, and they're cheap to run on every change. I only reach for an LLM judge for the fuzzy stuff like tone or completeness. One thing I'd add to the workflow: re-run your eval set on a schedule, not just on prompt changes. I run mine every six hours against the deployed chat, because data drift breaks prompts just as quietly as edits do.
I like your idea, but to avoid getting the same score every six hours (considering you run only code checks), are you dynamically adding new items to your evals dataset?
Good question. The six-hour checks are intentionally a fixed set, consistency is the point there. Same questions, watch for drift. The fuller eval set is the one that grows: 80 questions across three runs, and I add to it whenever new content ships or a new failure mode shows up in production. So the scheduled run tells me "did anything move," and the growing set tells me "are we actually getting better." Both matter, they just answer different questions.
The auto-closed issues seem worth a separate check: an unchanged overall score could hide a new false DUPLICATE if another case improves. I’d report those regressions separately from placement errors, and sample a few auto-closed issues for human review so new failure patterns can enter the dataset. Do you currently audit that path, or is detection mostly through the saved cases?
Leave-one-out replay is the part most eval write-ups skip. I run a much smaller version on my agent memory: 20 saved cases, checks decided by code instead of a model, and the nightly score compared with the previous run. 20 is tiny, but diffing against last night is what makes quiet degradation visible. Agree the false DUPLICATE is the one to weight most, since the auto-close means nothing downstream ever flags it.
Thanks for having me, it was great to be able to demo the FAQ!
Love our chats, Alexey! See you next time, either at Decoding AI, DataTalksClub, or personally in Berlin 😆
Alexey's leave-one-out trick for FAQ evals is a neat fix for the cold-start problem. The curation step that returns NEW, UPDATE, DUPLICATE or WRONG_COURSE is also a natural fit for a decision model rather than a chat model: fixed labels, a confidence score, and your code decides what happens next. I've been collecting guides and experiments for that pattern on JevMade.
Yes, decision models are a great fit for this use case! He implemented this before they existed. Still, not sure if they are cheaper than a gpt nano
This matches how I ended up working too. I build a RAG assistant for a content platform, and the thing that actually kept me honest was starting with deterministic checks before bringing in any LLM judge. Simple rules like "every claim needs a cited source" and "flag answers below a support threshold" catch most regressions, and they're cheap to run on every change. I only reach for an LLM judge for the fuzzy stuff like tone or completeness. One thing I'd add to the workflow: re-run your eval set on a schedule, not just on prompt changes. I run mine every six hours against the deployed chat, because data drift breaks prompts just as quietly as edits do.
I like your idea, but to avoid getting the same score every six hours (considering you run only code checks), are you dynamically adding new items to your evals dataset?
Good question. The six-hour checks are intentionally a fixed set, consistency is the point there. Same questions, watch for drift. The fuller eval set is the one that grows: 80 questions across three runs, and I add to it whenever new content ships or a new failure mode shows up in production. So the scheduled run tells me "did anything move," and the growing set tells me "are we actually getting better." Both matter, they just answer different questions.
The auto-closed issues seem worth a separate check: an unchanged overall score could hide a new false DUPLICATE if another case improves. I’d report those regressions separately from placement errors, and sample a few auto-closed issues for human review so new failure patterns can enter the dataset. Do you currently audit that path, or is detection mostly through the saved cases?
Leave-one-out replay is the part most eval write-ups skip. I run a much smaller version on my agent memory: 20 saved cases, checks decided by code instead of a model, and the nightly score compared with the previous run. 20 is tiny, but diffing against last night is what makes quiet degradation visible. Agree the false DUPLICATE is the one to weight most, since the auto-close means nothing downstream ever flags it.