Agent Evals 101: Building Your First Evaluation Suite

Part of the How do you know your AI agents are doing their job series.

What you will have by the end of this

By the time you finish reading, you will know how to assemble a golden test set from cases you already have, run a structured LLM-as-judge pass over agent outputs, connect that to a production sampling loop, and describe what a minimum viable eval suite looks like in week one versus month three.

If you have not yet read how to know whether your agents are doing their job, start there. That pillar explains the measurement problem at the strategic level. This article is the hands-on companion. It assumes you have at least one agent either in development or recently live, and that nobody at your company has done this before. If you are still picking your first use case, bookmark this and come back once you have a task defined.

Getting an agent live for the first time surfaces a consistent gap: teams invest in prompts and integrations but ship without any automated way to tell whether the agent is producing correct outputs. According to a November 2025 LangChain survey, 52.4 percent of organizations run offline evaluations on test sets, and 37.3 percent conduct online evaluations in production. That gap between 52 and 100 percent is where agents quietly degrade.

What an agent eval actually measures

An LLM benchmark measures a model’s performance on a standardised academic task. An agent eval measures whether your agent completes a specific business task correctly, end to end, given the tools, retrieval, and context your production system provides.

The distinction matters because an agent can use a capable model and still fail your task. A routing agent might call the right tool but write the output to the wrong queue. A document synthesis agent might retrieve the right passages but hallucinate a figure in the summary. Neither failure shows up on a model leaderboard.

Agent evals have two layers:

• Task-level pass/fail. Did the agent complete the task? For a ticket triage agent, did it assign the correct priority and owner? This is binary and automatable.
• Quality scoring. Where the task was completed, how well? For a customer communication agent, was the tone appropriate and the content accurate? This requires a rubric and usually an LLM judge or a human reviewer.

Both layers matter. Pass/fail catches outright failures. Quality scoring catches the slow drift where an agent keeps completing tasks but the outputs get progressively worse.

Building your first golden set

A golden set is a collection of historical cases with known correct outputs. You annotate each case with the expected result, then run your agent against it and compare.

For most teams, the fastest source is closed support tickets, completed workflows, or past decisions that a human made and recorded. If your agent handles engineering ticket triage, as Gelato’s Gemini-based agent does (ticket accuracy moved from 60 to 90 percent after deployment), your golden set starts with tickets where the correct routing is already documented.

Aim for 30 to 50 cases in week one. Skew toward the failure modes you already know about: edge cases, ambiguous inputs, and the scenarios that caused the most rework before the agent existed. Add at least five cases where the correct answer is to do nothing or escalate, because agents that never abstain will fail those silently.

Annotate each case with:

1. The input, exactly as the agent will receive it.
2. The expected output or decision, with enough specificity to score against.
3. Any constraints the output must satisfy (format, field values, tone level).

Keep annotations in a version-controlled file. Your golden set is a codebase asset, not a spreadsheet you email around.

LLM-as-judge: how it works and where it breaks

For quality scoring at scale, you configure a second LLM call that receives the agent’s output, the expected output, and a structured rubric, then returns a score and a short rationale. The judge is not measuring the model; it is measuring whether your agent’s output meets your criteria.

A structured output schema forces the judge to produce machine-readable scores rather than prose opinions. A minimal schema has three fields: score (integer, 1 to 5), pass (boolean, true if score is 3 or above), and reason (string, one sentence). You can extend this with dimension scores for accuracy, completeness, and format compliance.

Known failure modes of LLM-as-judge:

• Positional bias. Some judges rate longer outputs higher regardless of quality. Randomise the order of outputs when you present alternatives for comparison.
• Sycophancy on rubric language. If your rubric uses the same phrasing as the agent’s prompt, the judge scores outputs that echo the prompt more highly. Write rubric criteria in different words than the system prompt.
• Domain blindness. A general-purpose judge will miss factual errors in specialist domains. For financial or medical tasks, add a factual-check step that queries a reference source or uses a domain-specific model.
• Calibration drift. The judge’s scoring distribution shifts when you change the model version behind it. Re-calibrate against human-scored examples after any model update.

Offline evals before launch, online evaluation in production

flowchart TD
    A[New agent version] --> B[Run against golden set]
    B --> C{Pass/fail threshold met?}
    C -- No --> D[Debug and revise]
    D --> B
    C -- Yes --> E[Deploy to production]
    E --> F[Sample production traces]
    F --> G[LLM-as-judge scoring]
    G --> H{Score below threshold?}
    H -- Yes --> I[Flag for human review]
    H -- No --> J[Log to dashboard]
    I --> K[Add to golden set]
    J --> L[Monitor for drift]

Offline evaluation is a regression gate. Before you deploy a new version of your agent, you run it against your golden set and require it to meet a minimum pass rate. A common starting threshold is 90 percent task-level pass and an average quality score of 3.5 out of 5. If the new version drops below either number, it does not ship.

Online evaluation samples a percentage of real production traces, typically 5 to 10 percent, and scores them asynchronously. This is where you discover failure modes your golden set did not cover. Every case that fails online scoring is a candidate for the golden set, which is how the suite grows over time.

The two modes are not interchangeable. Offline evals are fast and deterministic. Online evals are slower and messier but reflect actual usage. You need both. Morgan Stanley’s document retrieval agent moved document discovery from 20 to 80 percent across advisor teams, a result that required ongoing measurement in production to confirm and maintain, not just a pre-launch test.

What your eval suite looks like in week one versus month three

flowchart TD
    W1[Week one] --> A1[30 to 50 golden cases]
    W1 --> A2[Task-level pass/fail automated]
    W1 --> A3[LLM-as-judge on quality, human spot-check 20 percent]
    W1 --> A4[Manual review of all failures]

    M3[Month three] --> B1[150 or more golden cases including production failures]
    M3 --> B2[Offline eval in CI pipeline]
    M3 --> B3[Online eval sampling 5 to 10 percent of traces]
    M3 --> B4[Dashboard with score trends and regression alerts]
    M3 --> B5[Human review focused on low-confidence judge outputs]

In week one, you are building the habit and proving the infrastructure works. You do not need a dashboard. You need a script that runs your agent against your golden set, calls your judge, and prints a pass/fail report.

By month three, that script lives in your CI pipeline. Every pull request that touches agent logic runs the full golden set before merge. A production sampling job runs nightly, pipes failures into a review queue, and logs scores to a time-series store so you can see trends. Teams that operate this way can catch regressions in hours rather than weeks. Klarna’s customer service agent, which handled the equivalent work of 853 full-time agents across 23 markets, required exactly this kind of continuous measurement to operate at that scale without service quality falling.

For agent observability tooling, the market now includes both dedicated eval platforms and observability layers built into frameworks. Tools like Prefactor sit in the category of platforms that combine trace capture with structured evaluation workflows, which is useful if you want the sampling and scoring pipeline in one place. The underlying approach, capturing traces, scoring with a rubric, routing failures to review, is the same regardless of tooling.

For more on the scoring and rubric design side, the LLM evaluation and agent evaluation pages cover the landscape of approaches. If you are building out the governance layer around your evals, AI governance best practices connects measurement to the broader accountability structure.

Where to start

If you have historical cases with known outcomes, you can build your first golden set this week. Run your agent against it, add a simple LLM-as-judge call with a three-field schema, and you have the core of an eval suite. Take the agent readiness assessment to identify which gaps in your current setup, including evals, to close before your next deployment.