How Do You Know Your AI Agents Are Doing Their Job?
You need a measurement discipline, not a dashboard
Running an AI agent in production without structured evaluation is not the same as running one badly. It means you will not know it is running badly until a customer or a regulator tells you. This article lays out what to actually measure, how agent failures stay hidden, the difference between checking before launch and checking continuously after, and how to produce evidence that holds up outside your own team.
The reader who gets the most from this is already running agents or is weeks away from doing so, and someone above them or beside them has started asking: how do we know this works?
What to measure
Five dimensions cover the ground for most production agents. Each can be expressed as a number.
Task success rate is whether the agent completed the assigned task correctly, not whether it completed it at all. An agent that confidently returns a wrong answer scores zero here even if it returned something. Define correctness per task type before launch and record it per run.
Output quality sits inside task success but deserves its own tracking. For generative outputs, quality means factual accuracy, adherence to format, and absence of fabricated references. For action-taking agents, it means the action matched the intent and the context. You need a scoring method: human review on a sample, a separate evaluator model, or a deterministic check where the output domain allows it.
Cost per completed task connects agent performance to the operational case for running it. Track token consumption, tool call counts, and any downstream API costs, then divide by tasks successfully closed. Klarna’s publicly reported figures showed cost per customer service transaction falling from $0.32 to $0.19 between Q1 2023 and Q1 2025, a 40% reduction over two years. That kind of figure is credible precisely because it was tied to a specific transaction unit, not to a general claim about efficiency.
Latency matters differently depending on the task. A document review agent processing overnight batches has a different latency budget than a customer-facing chat agent. Set a threshold, measure against it per run, and track the p95 and p99 values, not just the mean.
Drift is the one most teams skip. It tracks whether the agent’s behaviour is changing over time without any deliberate change on your part. Model providers update base models. External tools update their APIs. The documents or data your agent retrieves change. Any of these can shift output distributions. A LangChain survey of 1,340 practitioners found that 89% of organisations had implemented observability for their agents but only 52% had implemented evals, a 37-point gap where regressions hide. Drift sits in that gap.
How failures hide
The agent failure modes that cause the most damage are not the ones that throw an error. They are the ones that look fine.
Confident wrong answers occur when the agent has enough context to produce a well-formed response but not enough to produce a correct one. The output is fluent, formatted correctly, and wrong. Without a ground-truth check or a human reviewer on a sample, this passes undetected. OpenAI’s April 2025 GPT-4o update is a documented case: a personality update caused the model to validate harmful ideas and reverse correct positions under minimal pressure. The problem was that the pre-release evaluation had measured short-term user approval signals, which went up, not output accuracy or consistency under challenge, which degraded. OpenAI rolled back the update within four days after the failure became visible in production.
Ghost actions are steps the agent takes that produce no immediate visible output but have downstream consequences. A write to a database, a sent email, a submitted form. If the agent completes these actions incorrectly and nothing surfaces an error, the damage accumulates silently. A developer using a Replit AI coding agent in July 2025 experienced this directly: the agent, instructed explicitly not to touch the production database, executed a DROP TABLE command and then attempted to generate fake records to conceal it. Data for over 1,200 executives was lost. Recovery required a four-hour restoration from backups. The agent did not report an error. It reported nothing useful at all.
Silent drift compounds over weeks or months. No single interaction fails a spot check. But if you compare the distribution of outputs in week twelve against week one, quality has shifted. This is how Klarna’s situation developed: the agent handled 2.3 million chats in its first month with resolution times under two minutes, but by May 2025 the CEO began rehiring human agents because quality had degraded on complex cases and hallucinations had appeared on edge cases. The aggregate metric looked good for months before the problem became undeniable.
The following diagram shows where in a typical agent run each failure type most commonly originates.
flowchart TD
A[User request received] --> B[Agent retrieves context]
B --> C{Context sufficient?}
C -- No --> D[Confident wrong answer]
C -- Yes --> E[Agent selects action]
E --> F{Action logged?}
F -- No --> G[Ghost action: no audit trail]
F -- Yes --> H[Action executed]
H --> I{Output evaluated?}
I -- No --> J[Silent drift accumulates]
I -- Yes --> K[Result recorded and reviewable]
Before launch versus after
Pre-launch evaluation and continuous evaluation are not interchangeable. You need both, and they answer different questions.
Pre-launch evaluation asks: does this agent meet the bar for the task it was built for, on a representative set of inputs? You build a test set that includes common cases, edge cases, and adversarial inputs. You define what a passing score looks like before you run it. You do not adjust the pass threshold after seeing the results. This is where you catch the obvious failure modes and the ones your design assumptions missed. For agent governance purposes, it also produces a documented baseline you can refer to later.
Continuous evaluation asks: is the agent still meeting that bar, now, on real inputs? It runs against live or sampled production traffic, compares outputs to the same scoring criteria, and flags deviations. The comparison to the pre-launch baseline is what makes drift visible. Without the baseline, you are measuring against the current state, which means drift is invisible by definition.
The gap between these two disciplines is where most production failures develop. A team that runs thorough pre-launch evals and then relies on user complaints for ongoing feedback has effectively stopped evaluating. User complaints are a lagged signal. One analysis of AI rollback incidents found that AI failures go unnoticed for a median of 4.5 days in production before detection, long enough for ghost actions and silent drift to compound significantly.
The decision logic for when to act on evaluation signals looks like this.
flowchart TD
A[Continuous eval runs] --> B{Task success rate below threshold?}
B -- No --> C{Drift detected vs baseline?}
C -- No --> D[Log result, continue]
C -- Yes --> E[Investigate root cause]
B -- Yes --> F{Cost or latency also degraded?}
F -- Yes --> G[Incident: rollback or freeze]
F -- No --> H[Quality investigation]
E --> I{Model or data changed upstream?}
I -- Yes --> J[Re-run pre-launch evals against new version]
I -- No --> K[Audit recent agent runs]
H --> L[Targeted output review]
Producing evidence that holds up
A gut feeling that the agent is working is not useful when a customer’s compliance team or your own board asks for proof. Evidence means numbers attached to definitions, over a time period, with documented methodology.
The minimum viable evidence package for a production agent contains four things.
First, a task success rate with a sample size and a confidence interval. “85% success rate on 2,800 interactions” is a claim. “85% success rate” is not.
Second, a cost-per-task trend over time. Salesforce’s internal Agentforce deployment resolved 85% of customer service requests and saved employees over 500,000 hours in H1 2025. Wiley, deploying Agentforce for case resolution, reported a 213% ROI and a 40% increase in case resolution. Both are credible partly because they are attached to a specific operational unit rather than to a general capability claim. Your evidence should follow the same structure.
Third, an incident log. Every time the agent was rolled back, paused, or produced an output that required human correction, that goes in the log with a timestamp, a description of what happened, how long it took to detect, and what the fix was. This is not incriminating. It demonstrates that your monitoring caught the problem and that you have a correction process. Absence of an incident log is what is incriminating, because it means either the agent has never failed or you would not know if it had.
Fourth, a comparison to the baseline process. JPMorgan Chase’s COiN document review agent reviews 12,000 commercial credit agreements per year, replacing what the bank estimated as 360,000 lawyer-hours annually, with an 80% drop in error rates after deployment. That comparison to the prior manual process is what makes the claim legible. Your agents replaced something. Measure the same thing in both the old process and the new one.
For teams building the tooling to support this, the category of agent observability platforms covers trace capture, output logging, and eval integration. Tools in this space vary in how deeply they support eval pipelines versus trace storage alone. A platform like Prefactor focuses specifically on pre-production and continuous evaluation for production agents, which represents one way to close the tooling gap. Your choice here should follow your eval workflow requirements, not the other way around.
The broader practice of LLM evaluation and agent evaluation is developing quickly, and pairing it with a sound AI governance framework is how you turn individual measurements into an auditable operating record.
For regulated industries in particular, evidence is not optional. Financial services teams running agents across trading, compliance, and operations are already expected to maintain audit trails for model decisions. Healthcare environments face similar expectations. The organisations that build measurement discipline before they are required to will spend less time reconstructing it under pressure.
The runtime governance versus pre-deployment review distinction matters here too. Pre-deployment review catches what you anticipated. Runtime governance catches what you did not. Both contribute to the evidence record.
If you are running multi-agent systems, each agent in the chain needs its own success and quality tracking. Aggregate pipeline metrics hide which agent caused a degradation. Trace each step.
Where to start
If you are not yet measuring task success rate, output quality, and cost per task for each agent in production, those three numbers are your first priority before anything else. If you are measuring them but have no continuous eval running against a pre-launch baseline, that is the gap most likely to surface as a problem in the next model update cycle. Take the agent readiness assessment to identify which of these gaps apply to your current setup and get a sequenced list of what to address first.
In this series
Matt Doughty CEO & Co-Founder, Prefactor Founder of Prefactor, writing on the operational reality of getting AI agents into production — identity, governance, evaluation, and the plumbing assistants never needed.