How to Audit AI Agent Sessions for Hidden Bugs and Design Failures

What this guide gives you

You will leave with a repeatable process for reviewing AI coding agent sessions after they complete, a checklist of the failure categories worth looking for, and a decision framework for deciding which sessions to audit first. The process applies to any coding agent that produces a transcript, and it is grounded in two specific findings from this week: the release of Gage, an open-source tool that scans Claude Code transcripts for issues, and a study showing that Claude Code achieves an 84% pull request merge success rate in autonomous deployment, which also confirmed that frontier models can identify a vulnerability in their reasoning and still fail to act on it.

If you are still deciding how to measure agent performance in general, start with the pillar on how to know your agents are doing their job before working through this guide. And if your team is still designing the evaluation workflow itself, the guide on how to evaluate AI agents covers the upstream setup.

Why session audits are necessary

An 84% merge success rate sounds strong until you account for scale. Google reported that AI-written code rose from 30% of its output in April 2025 to 75% by 2026, with one complex migration completing six times faster year over year. At that volume, a 16% failure rate is not a rounding error. It is a continuous source of regressions, security gaps, and technical debt that accumulates faster than engineers can review it manually.

The core problem is not that agents fail. It is that the failures are not random. Agents tend to fail in clusters: the same class of mistake repeats across sessions because the underlying cause is a prompt design flaw, a missing constraint, or a tool configuration that produces ambiguous results. Session auditing is how you find those clusters before they reach your users.

86% of organizations have moved beyond experimentation and are deploying AI coding agents for production code, with enterprises leading at 91%. Most of them do not yet have a systematic way to learn from what went wrong.

The four failure categories to look for

Every session audit should check for failures in four areas.

Bugs introduced. Code that compiles and passes the agent’s own tests but contains logic errors, off-by-one conditions, or incorrect assumptions about upstream data. These are the hardest to catch because the agent typically reports success.

Missed red flags. Cases where the transcript shows the agent identified a problem, noted it, and then proceeded anyway. The 84% merge success study documented exactly this pattern: the model flagged a potential vulnerability in its chain of reasoning and did not act on the flag. Searching transcripts for phrases like “however,” “although,” or “this may cause” followed by continued execution is a reliable heuristic for surfacing these moments.

Design failures. The agent completed the task as specified but the specification itself was wrong or incomplete. The agent had no way to know, and the transcript will show clean execution toward a bad outcome. These require a human to read the goal alongside the output.

Tool misuse. The agent called the right tool with the wrong parameters, or called tools in an order that produced a race condition or partial state. This shows up in the tool-call log, not in the prose of the transcript.

A workflow for prioritizing which sessions to audit

You cannot audit every session at production scale. The following decision flow helps you triage.

flowchart TD
    A[Session completes] --> B{PR merged?}
    B -- No --> C[Low priority queue]
    B -- Yes --> D{Regression reported within 72h?}
    D -- Yes --> E[Immediate full audit]
    D -- No --> F{Session touched auth, payments, or data access?}
    F -- Yes --> G[Priority audit within 24h]
    F -- No --> H{Random sample trigger?}
    H -- Yes --> I[Standard audit queue]
    H -- No --> J[Log and skip]

The random sample trigger in the final branch should fire on roughly five to ten percent of sessions. That rate gives you a statistically useful baseline without overwhelming your review capacity.

How to structure the audit itself

Step 1: Retrieve the full transcript

A session audit is only as good as the record it reads. Store raw transcripts, including tool calls and their responses, at session close. If your agent infrastructure does not do this automatically, configure it before your next deployment. Agent observability tooling can handle transcript capture as part of a broader logging setup.

Step 2: Run automated scanning

Automated scanning finds the low-hanging failures: known anti-patterns, tool misuse signatures, and the “identified but ignored” flag pattern described above. The open-source Gage tool scans Claude Code transcripts and surfaces these categories without requiring you to build your own parser. If your agents run on a different substrate, the same pattern applies: define a set of regular expressions and semantic queries against the transcript and run them on ingest. Some teams use a secondary language model call for this step. Tools like Prefactor take a similar approach for structured agent review.

Step 3: Human review of flagged sessions

Automated scanning produces a shortlist. A human reviewer reads that shortlist and classifies each finding as a bug, a missed red flag, a design failure, or a false positive. This step should take fifteen to thirty minutes per flagged session if the transcript is well structured. If it is taking longer, the transcript format is the problem, not the reviewer.

flowchart TD
    A[Raw transcript] --> B[Automated scanner]
    B --> C{Findings above threshold?}
    C -- No --> D[Archive session]
    C -- Yes --> E[Human reviewer]
    E --> F{Finding classified?}
    F -- Bug --> G[File issue, link to session]
    F -- Missed red flag --> H[Update prompt constraints]
    F -- Design failure --> I[Revise task specification]
    F -- False positive --> J[Update scanner rules]

Step 4: Feed findings back into your evaluation baseline

A finding that does not update something is a finding that will recur. Route bugs to your issue tracker with a link to the session transcript. Route missed red flags to your prompt engineering review. Route design failures to whoever owns the task specification. Track the classification counts per week so you can see whether a particular failure category is growing.

The Alberta Government ran 50 agents across 466 million lines of code in 20 hours to rebuild government services across 27 ministries. At that scale, the cost of a missed systematic failure is not one bad PR, it is a pattern baked into dozens of services simultaneously. A feedback loop from audit findings to prompt and specification updates is what keeps that pattern from propagating.

Connecting audits to broader governance

Session auditing is one layer of a complete AI agent governance posture. It works alongside pre-deployment review, runtime controls, and access policies. If you are building out that broader posture, runtime governance versus pre-deployment review explains where each layer fits and what it cannot cover on its own. For teams that have security responsibilities alongside reliability ones, AI security best practices covers the threat surface that audit workflows need to account for.

LlamaIndex reached a state where 95% of its codebase is AI-generated. At that ratio, the audit process is not a quality gate bolted onto a human workflow. It is the primary quality mechanism. The teams getting there fastest are the ones treating session transcripts as first-class engineering artifacts, storing them, indexing them, and learning from them systematically.

Where to start

Run a retrospective audit on the last ten merged PRs your coding agent produced, using the four failure categories above as your checklist. Note which category appears most often. That is your first prompt or specification fix. To get a clearer picture of where your agent program stands overall, take the agent readiness assessment.