What is a chaos engineering harness for AI agents?

It is a test environment that deliberately injects realistic failures, expired tokens, rate limit responses, malformed payloads, and partial tool-chain errors, against your agent before it reaches production. The goal is to surface integration failures under controlled conditions where you can observe and fix them.

Why do unit tests miss API boundary failures?

Unit tests run against mocked dependencies that always behave correctly. The actual failure modes, token expiry mid-session, a third-party API returning a schema that changed overnight, a rate limiter that only triggers under concurrent load, only appear when the agent calls the real or realistically faked endpoint.

How often should I run chaos injection tests?

Run them on every build that touches a tool definition, a prompt, or an upstream dependency version. Agents without automated evaluations running on every prompt change had a 47% rollback rate over the prior year, versus 9% for agents with full evaluation coverage, according to a 2026 Forrester Enterprise AI Panel.

Do I need a dedicated chaos engineering platform to do this?

No. A proxy layer that intercepts tool calls and returns configured failure responses is enough to start. You can build this in a few hundred lines with any HTTP interception library. Dedicated platforms add fault scheduling and coverage reporting, which are worth adding once your basic fault library is in place.

Testing Agent Tool Calls Before They Break Production: Chaos Engineering for Agent Integrations

What you will build and why it matters

This guide shows you how to construct a chaos engineering test harness that intercepts your agent’s tool calls and injects realistic API failures before any real user triggers them. By the end, you have a repeatable fault library covering expired tokens, rate limit blocks, schema mismatches, and cascading tool-chain failures, and a clear signal about which failures your agent handles gracefully and which ones it does not.

The motivation is numerical. According to analysis of 847 documented agent deployments, 62% of AI agent deployment failures involved authentication issues. Separately, Gartner attributes 68% of AI project failures to legacy system integration and API integration issues. These are not exotic edge cases. They are the normal failure modes of any system that crosses an API boundary, and agents cross many of them in sequence.

If you are new to how agents reach those boundaries in the first place, how AI agents work covers the underlying mechanics before you build tests around them.

Why the API boundary is where agents actually fail

An agent does not fail because its reasoning is wrong. It fails because the environment it reasons about behaves differently from the environment it was tested against. The tool call is the point where those two environments meet.

Consider what happens during a single agent turn. The agent decides to call a tool, formats a request, receives a response, and uses that response to decide what to do next. If the response is a 401 because the session token expired after 55 minutes, or a 429 because three concurrent agent instances hit the same endpoint, or a 200 with a JSON schema that changed after a vendor update, the agent’s next decision is based on a situation its developers never modeled.

JPMorgan Chase, when deploying 500 or more AI agents across 250,000 employees via its LLM Suite, explicitly addressed security concerns around long-running agents and authentication stability before production rollout. The agents now generate investment banking deliverables in 30 seconds and contribute to $2 billion in annual savings, but clearing integration testing was a precondition, not an afterthought.

UCHealth deployed vital sign monitoring agents across 14 hospitals and 22,000 beds, integrating with medical device APIs, EHR systems, and alert APIs. The agents detect sepsis two to four hours earlier than previous methods. That outcome depends entirely on notification API calls completing reliably. A timeout in that chain is not a degraded experience; it is a missed alert.

Building the harness: four components

flowchart TD
    A[Agent under test] --> B[Fault injection proxy]
    B --> C{Fault rule match?}
    C -- Yes --> D[Return configured failure]
    C -- No --> E[Forward to real or stub API]
    D --> F[Agent response handler]
    E --> F
    F --> G[Assertion layer]
    G --> H[Pass / Fail / Retry log]

1. The fault injection proxy

Place an HTTP proxy between your agent runtime and every upstream tool endpoint. The proxy holds a fault rule table: a list of endpoint patterns, trigger conditions, and the response to return. A rule might say: for any POST to /auth/token, return a 401 with body {"error": "token_expired"} on the third call within a session.

The proxy does not need to be complex. An interception library in Python, Node, or Go, with a small rule engine reading from a YAML or JSON config file, is sufficient. The important property is that rules are version-controlled alongside your agent code so fault coverage evolves with the tool set.

For multi-agent systems, you need the proxy to handle fan-out: one orchestrator call can trigger several downstream tool calls in parallel, and you want to inject failures into specific legs of that fan-out independently.

2. The fault library

Build a catalogue of failure types before you write a single test. Common categories:

Authentication failures: expired tokens, revoked API keys, OAuth flows that time out mid-redirect.
Rate limiting: 429 responses with and without Retry-After headers, quota exhaustion on per-minute and per-day limits.
Schema drift: responses where a required field is missing, a field type has changed from string to integer, or an array is returned where the agent expects an object.
Partial tool-chain failure: the first tool in a sequence succeeds, the second returns a 503. Does the agent retry the whole chain, only the failed step, or stop and surface an error?
Latency spikes: responses delayed by 10 to 30 seconds to test timeout handling and whether the agent retries before its caller does.

Salesforce, running Agentforce across 18,000 or more customers and handling 2 million customer conversations, built real-time observability tracking latency, escalations, and errors at the API boundary on every prompt change. That data informed which fault types to prioritise in their test coverage.

3. Assertions and acceptance criteria

A chaos test without an assertion is just noise. For each injected fault, define what correct agent behaviour looks like. The agent should either retry with exponential backoff and succeed, surface a structured error to the caller, or escalate to a fallback tool, depending on what the fault type warrants.

flowchart TD
    A[Inject fault] --> B[Run agent turn]
    B --> C{Expected behaviour?}
    C -- Retry and recover --> D[Assert retry count and delay]
    C -- Structured error --> E[Assert error schema]
    C -- Escalate to fallback --> F[Assert fallback tool called]
    D --> G[Record result]
    E --> G
    F --> G
    G --> H{All assertions pass?}
    H -- Yes --> I[Green]
    H -- No --> J[Fail build]

Cohere Health processes 40,000 prior authorizations per day through agents that integrate with payer APIs, provider portals, and legacy auth workflows. At that volume, a single unhandled schema mismatch propagates across thousands of decisions before a human notices. Their approach to API failure testing before production treats assertion coverage as a release gate, not a nice-to-have.

4. Running faults on every build

Fault injection tests only provide value if they run continuously. Wire the harness into your CI pipeline so it executes on every commit that touches a tool definition, a prompt template, or an upstream dependency version. Tag each test with the fault type and the tool it targets so you can track coverage over time.

This is also where agent observability in production connects back to the harness. When your monitoring surfaces a new failure pattern in live traffic, you add the corresponding fault to the library and write a regression test before the next release.

Tools like Prefactor sit in this space, providing fault scheduling and coverage reporting for agent tool calls, if you want a dedicated platform rather than a hand-rolled proxy.

For governance teams asking how this fits into a broader review process, the comparison of runtime governance versus pre-deployment review is worth reading alongside this guide.

What the harness does not cover

The harness tests whether your agent handles known failure modes correctly. It does not test whether the agent’s reasoning produces correct outputs, which is agent evaluation territory. It does not test whether your prompts are stable across model versions, which belongs in prompt engineering regression suites. And it does not replace monitoring after launch: SharkNinja, running a personal shopper agent through Agentforce to 250,000 conversations, found that data quality and API guardrail issues surfaced iteratively even after pre-launch validation. The harness reduces the rate of production surprises; it does not eliminate them.

For the security-specific angle on what happens when authentication failures are exploited rather than accidental, AI security best practices covers the overlap between chaos testing and threat modelling.

Where to start

Map every tool call your agent makes, identify the three failure types most likely to affect each one, and build the proxy and fault library around those first. Once your baseline passes, add the harness to your CI pipeline. Take the agent readiness assessment to see which other gaps between your current test coverage and a production-ready agent are worth closing next.