Is it safe to use AI agents with patient data (PHI)?

Only if the whole pipeline is covered. Every hop — model provider, tools, traces, evaluation logs — is a place PHI can land, and step-level observability traces are the commonly missed one. PHI redaction must happen at trace-write time, and every processor in the chain, including the logging vendor, needs a BAA under HIPAA.

When does a healthcare AI agent become a regulated medical device?

When it crosses from administrative support into clinical decision-making. Scheduling, intake, and documentation support is administrative; an agent that suggests codes, drafts clinical content, or influences care can fall under FDA software guidance and needs documented human accountability.

What controls do healthcare AI agents need?

Redact PHI at trace-write time under BAA, classify each agent as administrative or clinical-influencing, track edit and rejection rates on human review (near-zero rates trigger investigation, not celebration), evaluate coding-adjacent agents continuously against gold-standard cases with drift alerts owned in compliance, and give agents distinct EHR identities with minimum-necessary scopes.

healthcare

AI agent readiness in healthcare

Healthcare's agent adoption is concentrated where the industry bleeds money and staff time: documentation, prior authorisation, revenue cycle, and patient communication. The constraint is not ambition — it is that agents here operate next to PHI and clinical decisions, where a wrong action is measured in patient harm and federal penalties, not refunds. The newest pressure is agentic: as documentation and payer-facing workflows move from drafting to [acting](/agentic-ai) — filing, submitting, communicating without a human on every step — the readiness bar moves with them.

Adoption drivers

Specific risks

PHI exposure through the agent stack

Every hop in an agent pipeline — model provider, tools, traces, evaluation logs — is a place PHI can land. Observability data is the commonly missed one: step-level traces capture tool inputs and outputs, which in healthcare means they capture PHI unless redaction happens at write time. BAAs must cover the whole chain, including the logging vendor.

Clinical influence without clinical accountability

An agent that drafts a discharge summary or suggests a code is influencing clinical and billing records. When the human review step becomes a rubber stamp — and under volume pressure it does — the agent is effectively making decisions no licensed professional has owned. Review-rate and edit-rate metrics are the early warning.

Coding and billing drift

Agents that touch coding sit in False Claims Act territory. Systematic drift toward higher-value codes, even unintended, compounds into liability at claim volume. Continuous evaluation against gold-standard coded cases is the control, not periodic spot checks. Drift alerts also need a named owner in compliance — a dashboard nobody is paid to watch is not a control.

Regulatory context

Readiness checklist

PHI redaction happens at trace-write time, and the observability vendor is under BAA
Every agent is classified administrative vs clinical-influencing, and the second class has documented human accountability
Edit and rejection rates on human review steps are tracked; near-zero rates trigger investigation, not celebration
Coding-adjacent agents are evaluated continuously against gold-standard cases with drift alerts
Agent access to EHR systems uses distinct identities with minimum-necessary scopes
Patient-facing agents disclose themselves where state law requires