How long should I spend collecting baseline data before deploying an agent?

Four to six weeks of logged data covers most business cycles and gives you enough variation to set realistic targets. If your process is seasonal, capture at least one peak period in the baseline window.

What if the process I want to automate has never been formally measured?

Start with a two-week observation period where operators log time and error counts manually. Imperfect baseline data is still far more useful than none, because it gives you a fixed reference point to compare against after deployment.

Which metric matters most when reporting agent impact to leadership?

Pick the one metric that directly maps to a cost centre or revenue line your leadership already tracks, then report agent impact in those same units. Cycle time and error rate are useful internally, but finance teams respond to figures they can tie to budget lines.

Can I reconstruct a baseline from historical logs after the agent is already running?

Sometimes, if your ticketing or ERP system retains timestamped records. Pull data from a period at least three months before go-live, exclude any months where a process change occurred, and treat the reconstruction as approximate. A reconstructed baseline is better than none, but note its limitations when you report results.

How to Measure What Your Agent Actually Did (Before and After)

What this article gives you

You will leave here with a concrete method for capturing the state of a process before an agent touches it, and a matching set of metrics to run afterward. That before-and-after pair is what turns an agent deployment from an experiment into a business case.

The problem is common. 74% of major enterprises have AI in production, yet 50% cannot consistently measure ROI. Measurement is not a post-launch task you schedule when things settle down. It is a prerequisite, and the work begins before the agent runs a single task.

If you are still deciding whether your agents are working in a broader sense, the pillar how do you know your agents are doing their job covers the ongoing monitoring question in full. This article focuses on the narrower problem: what to record before you deploy, so the after-state means something.

Why baselines disappear before you think to capture them

Once an agent is running, the team’s attention shifts to debugging and iteration. The pre-deployment state of the process fades quickly, and with it the reference point you need. McKinsey’s August 2026 analysis found that enterprise AI investment is rising but reported impact on earnings remains flat, specifically because early adoption lacks outcome rigor. The investment is real; the proof is missing.

CIO’s reporting from the same month noted that automation projects frequently fail to establish baselines before implementation, leaving teams unable to quantify savings after the fact. The gap is not technical. It is procedural. Nobody assigned the measurement task before the build started.

The fix is to treat baseline capture as a deliverable with a deadline, the same way you treat the agent’s first deployment.

What to measure before the agent starts

The goal is to document the process as it actually runs today, not as the runbook describes it. Four categories of data cover most agent use cases.

1. Volume and frequency

Count how many instances of the task occur per day, week, or cycle. For a customer service triage agent, that is incoming ticket volume. For a supply chain agent, it is purchase order events. You need the distribution, not just the average, because agents often handle peaks differently than troughs.

2. Cycle time

Record the elapsed time from task arrival to task completion. Include waiting time, not just active processing time. If a contract review sits in a queue for two days before a human opens it, that two days belongs in your baseline. Salesforce’s internal legal team cut $5 million in legal costs using contract automation agents, but that figure only holds up because they had a prior cost structure to subtract from.

3. Error and rework rate

Log how often the output requires correction, escalation, or a second pass. This is the metric most teams skip, because it requires someone to admit that the current process has a failure rate. Record it anyway. An agent that reduces rework from 18% to 4% is delivering real value, but only if you measured 18% first.

4. Cost per unit

Translate the above into a cost figure your finance team already uses. Headcount hours multiplied by a burdened rate is the simplest version. If the process touches third-party fees, include those too. Klarna’s customer service agent did the equivalent work of 853 full-time employees and reduced repeat inquiries by 25%, saving $60 million annually. That figure is credible because it maps directly to a prior staffing cost.

The measurement window

flowchart TD
    A[Identify target process] --> B[Define 4 baseline metrics]
    B --> C[Log data for 4 to 6 weeks]
    C --> D{Enough variation captured?}
    D -- No --> C
    D -- Yes --> E[Freeze baseline snapshot]
    E --> F[Deploy agent in parallel or shadow mode]
    F --> G[Log same 4 metrics for agent]
    G --> H[Compare at 30, 60, 90 days]
    H --> I[Report delta to stakeholders]

Run your baseline logging window for four to six weeks before go-live. If the process has a weekly cycle, four weeks captures it. If it is monthly, extend to eight. The window closes when you freeze the snapshot, not when the agent launches.

After launch, log the same four metrics under agent operation. Compare at 30, 60, and 90 days. Thirty days catches early anomalies. Sixty days smooths them. Ninety days is the number you present to leadership.

Avoiding the confidence-without-evidence trap

Only 29% of executives can measure AI ROI with confidence, even though 79% report seeing productivity gains. The gap between “it feels faster” and “cycle time dropped from 4.2 hours to 1.1 hours” is the gap between anecdote and a budget renewal.

Three patterns create that gap.

Selecting metrics after the fact. If you choose what to measure once you have seen the agent’s output, you will unconsciously pick the metrics where it looks best. Define your four categories before the agent runs anything.

Measuring outputs instead of outcomes. An agent that closes 90% of support queries autonomously is doing well on output. Whether that translates to fewer escalations, lower cost per resolution, and higher first-contact resolution rates is the outcome question. Tripadvisor’s AI customer service agent reached 90% autonomous query resolution, but the useful business metric is what happened to cost and customer wait time alongside that figure.

Ignoring the counterfactual. Processes are not static. Volume grows, teams change, and seasonal effects shift the numbers. When you report agent impact at 90 days, note what the baseline would likely have looked like at that date under normal growth assumptions. A flat cycle time is a gain if volume grew 30% in the same period.

For teams running multi-agent systems or complex agentic AI workflows, attribution gets harder because several agents may touch a single task. In those cases, instrument each hand-off point separately so you can isolate which agent is contributing which effect. The agent observability practices that help with this sit alongside measurement but serve a different purpose: they tell you what the agent is doing step by step, while your baseline framework tells you whether the process is better.

Structuring the before-and-after report

flowchart TD
    A[Baseline snapshot] --> B[Volume per period]
    A --> C[Cycle time distribution]
    A --> D[Error and rework rate]
    A --> E[Cost per unit]
    F[90-day agent snapshot] --> G[Volume per period]
    F --> H[Cycle time distribution]
    F --> I[Error and rework rate]
    F --> J[Cost per unit]
    B --> K[Delta and % change]
    C --> K
    D --> K
    E --> K
    G --> K
    H --> K
    I --> K
    J --> K
    K --> L[Report to stakeholders]

The report structure is simple. Four rows, two columns: baseline value and current value. Add a percentage change column and a note on what drove each shift. If the agent is one of several changes that occurred in the period, say so. Honest attribution is more credible than a clean number that stakeholders will question later.

Teams using a tool like Prefactor to govern agent behaviour can pull execution logs directly into this report, since the platform records what each agent did at each step. Whatever tooling you use, the log format matters less than the discipline of capturing it from day one.

For broader guidance on evaluating AI agents beyond the financial case, including task accuracy and safety checks, the linked guide covers the full evaluation surface. If you are earlier in your deployment journey and want to understand how AI agents work before committing to a measurement framework, that is a useful prior read.

JPMorgan Chase’s 450-plus AI agents in production produced a reported 20% increase in gross sales from AI-driven market screening and client analysis. That is a credible figure because JPMorgan operates measurement infrastructure at scale. For most teams deploying their first agent, the equivalent discipline is a shared spreadsheet, a four-week logging window, and a meeting scheduled at day 90 to read the numbers out loud.

Where to start

Before you write a line of agent code or configure a single tool, open a document and write down the four baseline metrics for the process you are targeting: volume, cycle time, error rate, and cost per unit. Set a calendar reminder to freeze those numbers before go-live. If you want a structured way to check whether your organisation is ready to capture and act on this data, take the agent readiness assessment to identify where your measurement foundations need work before deployment begins.