How to Evaluate an LLM Feature Before It Reaches Customers: A Practical Eval Playbook

In November 2022, Jake Moffatt asked Air Canada's website chatbot how bereavement fares worked. The chatbot told him he could book a full-price ticket and apply for the bereavement discount afterwards. That was not the airline's policy. When he applied, Air Canada refused, and argued in part that the chatbot was responsible for its own statements. In February 2024 the British Columbia Civil Resolution Tribunal rejected that argument, found the airline liable for negligent misrepresentation, and ordered it to pay the fare difference plus interest and fees.

A month later, The Markup reported that New York City's MyCity business chatbot was telling small business owners they could take a cut of workers' tips and that landlords did not have to accept Section 8 vouchers. Both answers were wrong under city and state law. The city added disclaimers, kept the bot running as a pilot, and, according to The Markup's January 2026 follow-up, eventually took it down.

Neither failure needed a better model. Both needed someone to test the specific questions real users would ask, against the specific facts the system was supposed to know, before launch. That process is called evaluation, or "evals," and it is the most under-built part of most AI products.

This is the playbook we use to evaluate LLM features before they reach customers.

Why normal QA is not enough

Traditional software testing assumes that the same input produces the same output, and that "correct" can be written down in advance. LLM features break both assumptions.

• Outputs vary. The same question can produce differently worded answers, and occasionally a different answer.
• Inputs are open-ended. Users type things nobody anticipated. A test suite of 30 hand-written questions covers a small fraction of real traffic.
• Quality regresses silently. A prompt edit, a model version update or a change in the retrieved documents can make answers worse without causing a single error in the logs.

The industry knows this is the bottleneck. In LangChain's 2026 State of Agent Engineering survey of about 1,340 practitioners, quality was again the most-cited barrier to getting agents into production, ahead of cost.

A working evaluation process has five layers. Each catches failures the others miss.

Layer 1: Error analysis before metrics

The most common mistake is to start by choosing metrics ("faithfulness," "helpfulness," "toxicity") from a framework. Generic metrics measure generic problems. Your system's problems are specific.

Start instead by reading real outputs. Practitioners such as Hamel Husain, whose writing on evals is widely used in the industry, recommend this sequence:

• Collect 50 to 100 real or realistic traces. A trace is the full record of one interaction: user input, retrieved context, tool calls and final answer.
• Have one domain expert review each trace and write a short note on anything wrong. Not a score, a note. "Quoted the 2024 refund window, policy changed in March."
• Group the notes into failure categories. You will usually find five to ten categories, and two or three of them account for most failures.
• Count them. That count is your first real quality baseline.

This takes a day or two. It almost always reveals problems nobody predicted, and it tells you which automated checks are worth building. Fix the obvious problems you find before you automate anything.

Layer 2: Deterministic checks in code

Many failures can be detected without any judgment at all. These checks are cheap, fast and fully reliable, so they should run on every change.

Typical examples:

• The output parses as the required JSON schema.
• Every cited document ID exists in the retrieved context.
• Any dollar amount, date or percentage in the answer appears in the source documents.
• The answer does not contain phrases the business has banned (guarantees, legal advice, competitor names).
• Tool calls use valid arguments, for example a refund amount never exceeds the order total.

import re

def check_numbers_are_grounded(answer: str, context: str) -> bool:
    """Every number in the answer must appear in the retrieved context."""
    numbers = re.findall(r"\$?\d[\d,]*(?:\.\d+)?%?", answer)
    return all(n.strip("$%") in context for n in numbers)

def test_refund_policy_answers(cases):
    failures = []
    for case in cases:
        result = run_assistant(case["question"])
        if not check_numbers_are_grounded(result.answer, result.context):
            failures.append(case["id"])
    assert not failures, f"Ungrounded numbers in cases: {failures}"

A check like this would have flagged the Air Canada answer if the policy page had been the retrieved context: the chatbot told the customer he could claim the discount within 90 days of buying the ticket, and no such window existed in the policy.

Layer 3: LLM-as-judge, used carefully

Some qualities cannot be checked with code: whether an answer actually addresses the question, whether the tone is appropriate, whether a summary leaves out something important. For these, a second model can act as the grader.

The method has real research behind it. Zheng and colleagues (NeurIPS 2023) found that a strong model acting as judge agreed with human preferences more than 80% of the time, which is about the same rate at which humans agree with each other. The same paper documents the weaknesses: position bias (preferring whichever answer is shown first), verbosity bias (preferring longer answers), and self-enhancement bias (preferring answers written by the same model).

Used naively, an LLM judge gives you confident numbers that may not mean much. Used carefully, it scales expert judgment. The rules we follow:

• One judge per failure category from Layer 1, not one general "quality score." "Did the answer state a policy that is not in the provided documents? Yes or no." is gradeable. "Rate helpfulness from 1 to 10" is not.
• Binary pass/fail, with a written reason. Scales invite the judge to drift toward the middle.
• Validate the judge against human labels. Take 100 traces your domain expert already labeled, run the judge, and measure how often it agrees on both passes and failures. If it misses more than a small share of real failures, rewrite the judge prompt before trusting it.
• Swap answer order in pairwise comparisons and use a different model family for judging where possible, to reduce position and self-enhancement bias.
• Re-validate after every model change, including changes to the judge model.

Layer 4: Evaluate retrieval separately from generation

If your system answers from documents (retrieval-augmented generation), a wrong answer has two possible causes: the right document was not retrieved, or it was retrieved and the model ignored or misread it. These need different fixes, so measure them separately.

• Retrieval: for each test question, record which documents should be retrieved. Measure how often they appear in the top results (recall at k). If recall is low, no prompt change will fix the answers.
• Generation: given the correct documents, does the model answer faithfully?

Placement matters too. Liu and colleagues ("Lost in the Middle," TACL 2024) showed that models use information most reliably when it appears at the beginning or end of a long input, and noticeably less reliably when it sits in the middle, even in models built for long contexts. In practice this means retrieving fewer, better-ranked passages beats stuffing the context window, and the most relevant passage should not end up buried in position 7 of 15.

Layer 5: Production monitoring and the regression suite

Pre-launch evaluation tells you the system was good enough on the day you tested it. Production tells you what users actually do.

• Sample real traces every week and run them through the same error analysis as Layer 1. New failure categories appear as usage grows.
• Collect explicit feedback (thumbs down, "this was wrong") and route it to a review queue, not just a dashboard.
• Turn every confirmed failure into a test case. Over time this becomes your regression suite, and it is worth more than any public benchmark because it reflects your users.
• Run the suite as a release gate. No prompt change, model upgrade or retrieval change ships if it reduces the pass rate on known cases.

Where each check runs

The minimum bar before launch

If you only do part of this, do these five things before real customers use the feature:

• A domain expert has reviewed at least 100 realistic traces and the failure categories are written down.
• Deterministic checks exist for every failure that can be detected in code.
• Any LLM judge in use has been compared against the expert's labels.
• There is a written launch threshold, for example "no ungrounded policy statements in 200 test cases," agreed with the business owner.
• A process exists to turn production failures into new test cases.

The Air Canada decision is a useful reminder of why this matters. The tribunal held that the airline was responsible for all the information on its website, whether it came from a static page or a chatbot. Disclaimers help, but they do not replace testing.

The bottom line

Evals are not a research exercise or a feature you add after launch. They are the acceptance criteria for an AI system, the same way automated tests are for normal software. Teams that build them early ship faster, because every prompt change and model upgrade can be checked in minutes instead of argued about in meetings.

This is also one of the failure patterns behind stalled AI projects more broadly, which we covered in Why most AI projects stall before production.

Every system we deliver through our AI automation practice ships with its evaluation suite and regression gate, owned by the client. If you have an AI feature in pilot and are not sure whether it is ready for customers, get in touch and we can review it.

Sources

• Civil Resolution Tribunal of British Columbia, Moffatt v. Air Canada, 2024 BCCRT 149, February 2024; summary by McCarthy Tétrault
• The Markup, NYC's AI Chatbot Tells Businesses to Break the Law, March 29, 2024, and follow-up, January 30, 2026
• LangChain, State of Agent Engineering, 2026
• Hamel Husain, Your AI Product Needs Evals, March 2024
• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023
• Liu et al., Lost in the Middle: How Language Models Use Long Contexts, TACL 2024