How long does it typically take to go from a working demo to a production agent?

Most teams underestimate this by two to four times. A demo that impresses in a controlled environment usually needs two to four additional months of work on edge case handling, integration, permissions, cost controls, and observability before it is safe to run in production. The timeline depends on how many external systems the agent touches and how strictly regulated the domain is.

What is human-in-the-loop and when can you remove it?

Human-in-the-loop means a person reviews or approves the agent's output before it takes effect. You start there to build a baseline of measured performance, then reduce human involvement as the data shows the agent meeting agreed accuracy and error-rate thresholds. You remove it only for the specific task types where the evidence supports it, not for the whole agent at once.

How do you pick the right first use case for an AI agent?

Look for a task that is bounded in scope, produces outputs you can measure objectively, and has a failure mode your business can absorb. A task that handles thousands of low-stakes interactions a day is a better first candidate than one that handles ten high-stakes decisions a week, because volume gives you evaluation data faster and errors are individually cheaper.

What does Gartner say about agentic AI project failure rates?

Gartner predicts that over 40% of agentic AI projects will be canceled by end of 2027, citing escalating costs, unclear business value, and inadequate risk controls. The firm attributes much of this to projects that began as hype-driven experiments without a clear measure of what success looks like in production.

What It Actually Takes to Get Your First AI Agent Live

What you will get from this post

By the end of this post you will have a clear picture of why your proof of concept is not close to done, what work actually separates a demo from a live agent, and a concrete checklist to make the launch decision with confidence rather than pressure.

The gap between your demo and production is larger than it looks

Your proof of concept ran. It answered the right questions, called the right tools, and impressed the room. That is useful, but it tells you almost nothing about whether the agent will behave reliably when it meets real users, real data volumes, real permissions boundaries, and real edge cases.

Gartner predicts that over 40% of agentic AI projects will be canceled by end of 2027 due to escalating costs, unclear business value, or inadequate risk controls. The firm notes that most of these projects start as hype-driven experiments with no clear measure of what production success looks like. That is the trap. The demo proves the idea is possible. It does not prove the idea is ready.

The demo-to-production gap has five specific components, and each one requires deliberate work before you ship.

Edge cases. Your demo ran on curated inputs. Production will surface inputs you did not anticipate, incomplete data, ambiguous intent, multi-language text, and combinations of conditions your prompt never saw. Each one is a potential failure.

Integration. Calling a sandbox API is not the same as calling a production system with rate limits, authentication, versioning, and the possibility that the downstream service is temporarily unavailable. Every tool your agent uses in production needs a defined error path.

Permissions. What the agent is allowed to read, write, and trigger must be scoped explicitly. An agent that has access to more than it needs is a security and compliance risk, not a convenience.

Cost. A demo that runs ten conversations is not a cost model. An agent handling ten thousand conversations a day at variable token counts and tool-call frequencies needs a cost projection and a budget control before it goes live.

Latency. Users and downstream systems have latency expectations. If your agent takes 14 seconds to respond in a customer-facing context, you have a usability problem regardless of accuracy.

Klarna’s deployment is a useful reference for both sides of this. When the company launched its AI customer service agent in February 2024, it handled 2.3 million conversations in the first month, cut average resolution time from 11 minutes to under 2 minutes, and reduced cost per transaction from $0.32 to $0.19 over two years. Those are real, measurable production gains. But the same deployment also produced hallucinations on edge cases that raised compliance concerns and a drop in customer satisfaction scores on complex queries, which led the company to begin rehiring human agents by May 2025. The agent was genuinely ready for the high-volume, bounded cases. It was not ready, without human backup, for the long tail of complex ones.

Pick a first use case that can survive contact with reality

Before you close the demo gap, confirm you are working on the right use case. Many teams ship the demo they built rather than the one that was worth building.

A good first production use case has three properties.

It is bounded. The input space is well-defined, the tools the agent needs are limited, and the task has a clear completion state. “Answer questions about order status using our order management API” is bounded. “Help customers with anything they need” is not.

It is measurable. You can define correct and incorrect in advance, and you have enough volume to detect drift. An agent handling five hundred interactions a day gives you statistical signal in days. An agent handling three interactions a week gives you signal in months.

Its failure modes are tolerable. If the agent gets something wrong, what happens? A wrong product attribute gets corrected in the next catalog refresh. A wrong medical dosage recommendation does not. Your first agent should be in the category where individual errors are recoverable without serious harm, because errors will occur.

Wayfair chose product catalog enrichment as an early production use case: a bounded task, measurable output quality, and a failure mode where a wrong attribute gets corrected in the next update cycle. The result was product attributes updated five times faster with meaningful operational cost savings. That is what a well-scoped first use case looks like.

Use this decision flow before committing to a use case:

flowchart TD
    A[Candidate use case] --> B{Is the input space bounded?}
    B -- No --> C[Narrow the scope or pick another]
    B -- Yes --> D{Can you measure correct vs incorrect?}
    D -- No --> E[Define success criteria first]
    D -- Yes --> F{Is the failure mode tolerable?}
    F -- No --> G[Add human review or pick another]
    F -- Yes --> H[Proceed to production readiness work]

Instrument before you launch, not after

The most common production mistake is shipping an agent and then trying to understand what it is doing. You need observability in place before the first real user interaction, not as a follow-on task.

Agent observability means capturing, at minimum: every input the agent received, every tool call it made and the result, every output it produced, latency at each step, token counts and cost per conversation, and whether the conversation reached a successful completion state or escalated.

Without this data you cannot answer the most basic operational questions: Is the agent performing better or worse this week than last? Which input types are producing the highest error rates? Is a change to the underlying model affecting outputs? You cannot govern what you cannot measure.

Evaluation before launch means constructing a test set that represents your production input distribution, including edge cases you expect to be difficult, and running the agent against it with human-reviewed labels. This is not a one-time exercise. You run it against every significant change to the agent’s prompts, tools, or underlying model. Evaluating AI agents rigorously before each deployment is the difference between knowing your agent is ready and hoping it is.

United Wholesale Mortgage’s deployment of AI agents for underwriting document processing shows what instrumented, measured deployment looks like at scale. Working with Google Cloud, UWM more than doubled underwriter productivity from 6 loans per day to 14 loans per day in nine months, against an industry average of 3 loans per day. Underwriting is a domain with clear, auditable correctness criteria, which is precisely why it was possible to measure that gain with confidence. The measurement infrastructure was not an afterthought.

Start with human-in-the-loop and earn autonomy with data

The default posture for a first production agent is human review before consequential actions take effect. This is not a sign that the agent is not good enough. It is a risk control that lets you collect the data you need to justify expanding autonomy later.

Human-in-the-loop takes different forms depending on the use case. In a customer service context it might mean an agent drafts a response and a human approves it before it sends. In a document processing context it might mean the agent extracts and classifies fields, and a human reviews flagged low-confidence outputs. In a workflow automation context it might mean the agent proposes a set of actions and a human confirms before execution.

The goal is to reduce the human review burden progressively as the evidence supports it. You agree in advance on the thresholds: if the agent achieves X% accuracy on task type Y with fewer than Z escalations per thousand interactions over a thirty-day window, you remove human review for that task type. You do not remove it because the agent seems to be doing well. You remove it because the numbers say so.

This is how 1-800Accountant approached tax season: the company’s deployed agent autonomously resolved 70% of administrative chat engagements, with the remaining 30% handled by humans. The 70% figure is meaningful because it was earned against a measured baseline, not assumed from a demo. Pandora’s deployment reached a similar posture, with agents deflecting 60% of customer service cases autonomously and a 10% improvement in net promoter score, alongside human handling of the remainder.

The path from HITL to increased autonomy follows a consistent pattern:

flowchart TD
    A[Launch with full human review] --> B[Collect 30 days of labeled data]
    B --> C{Accuracy and error rate meet thresholds?}
    C -- No --> D[Identify failure patterns, retune, retest]
    D --> B
    C -- Yes --> E[Remove human review for that task type]
    E --> F[Monitor for drift]
    F --> G{Performance holding?}
    G -- Yes --> H[Consider expanding scope]
    G -- No --> I[Reinstate human review, investigate]

For AI governance purposes, document every threshold decision and every change to the human review scope. If something goes wrong, you need to show what controls were in place and when they changed.

The production readiness checklist

Before you make the launch decision, work through each of these. A “no” is a blocker, not a risk to accept.

Use case definition

The input space is defined and bounded
Success and failure criteria are written down and agreed with stakeholders
The failure mode has been reviewed and is tolerable without additional controls

Integration and permissions

Every external tool or API the agent calls has been tested under production authentication
Error paths are defined for every tool call, including timeouts and unavailability
The agent’s permissions are scoped to the minimum required for the task
A security review covering data access and output actions has been completed

Cost and latency

Token and tool-call costs have been projected at expected production volume
A cost ceiling and alerting threshold are in place
Latency at p50 and p95 has been measured under realistic load

Observability and evaluation

Input, output, tool calls, latency, and cost are logged per conversation
An evaluation set exists with human-reviewed labels covering normal and edge-case inputs
The agent has been run against the evaluation set and results are documented
A monitoring baseline is established so drift is detectable

Human-in-the-loop

The human review posture for launch is defined: what gets reviewed, by whom, and how fast
Escalation paths exist for cases the agent cannot handle
The thresholds for reducing human review are agreed in advance and documented

Governance and compliance

The agent’s behavior in regulated scenarios has been reviewed by the appropriate team
A rollback plan exists and has been tested: you can disable or revert the agent in under fifteen minutes
An AI governance framework covering this agent is in place or in progress

Tools like Prefactor sit in the observability and governance category, giving teams a single place to log agent traces, track evaluation results, and manage the thresholds that govern autonomy expansion. The category matters more than any specific tool: what you need is a system of record for agent behavior before you expand scope.

Where to start

If you have a proof of concept and pressure to ship, the most useful next step is an honest assessment of where you actually stand against the criteria above. Work through the checklist with the team that built the demo, not just the team that commissioned it. Take the agent readiness assessment to get a structured view of the gaps between your current state and a defensible production launch.

In this series

How to pick your first AI agent use case