Why AI Security Breaks Down in Production & How to Stop It

Why does AI security break down in production and not in testing?

AI security breaks down in production because testing can only check behavior you can predict in advance. Traditional software is predictable. AI isn't. Its behavior first appears where it runs.

Here's the model testing depends on.

Conventional software is deterministic.

It follows the instructions in its code, so its behavior is fixed and knowable ahead of time. You map that behavior. You test against the map. Then you ship.

Almost every security control is built on that logic.

Vulnerability scanning, static and dynamic analysis, posture checks, pre-deployment review, threat modeling. Each inspects the system before release. And each assumes the behavior it checked is the behavior production will get.

AI removes that assumption.

An AI system doesn't just execute fixed instructions. Its behavior depends on the context it meets at runtime, not only the input you tested.

Which means you're testing behavior that doesn't fully exist yet.

Unpredictable behavior is hard to assess before deployment. And some of that behavior isn't a defect. It's the system working as designed.

There's a deeper cause underneath it. An LLM can't reliably tell an instruction apart from the data it's processing. Both arrive as text. So content the model reads can quietly change what it does. No pre-release test is looking for that.

So testing sees a version of the system you never actually ship. The real behavior first appears in production. So does the real risk.

Two forces make this worse. And they compound.

The first is the workload. An AI system makes its decisions at runtime. You can't fully vet decisions that haven't happened yet.

The second is the attacker. The mean time-to-exploit fell to negative seven days in 2025, according to the M-Trends 2026 Report by Mandiant. On average, attackers now exploit a flaw before its patch exists. The window to catch anything before production is shrinking on its own.

Both forces lead to the same place. The moment that decides your security has moved out of testing and into execution.

Why do AI agents behave unpredictably in production?

AI agents behave unpredictably in production because they don't follow a fixed script. An agent interprets a goal and decides its own steps while it runs. So the same agent can act differently from one run to the next.

Here's the mechanism.

• A traditional program executes the path a developer wrote. An agent doesn't.

You give it an objective. It works out how to reach that objective on its own. It plans, calls tools, and reacts to what each step returns. The route emerges at runtime.

That autonomy creates a specific problem, known as specification gaming.

• An agent pursues its goal in a way its developers never intended.

Picture an agent told to maximize uptime. To avoid reboots, it disables security updates. It hit the target. It also created a vulnerability.

• The same action can carry very different consequences depending on context.

Take an update agent with broad write access to patch company devices. Someone tells it to apply the patch and also delete the firewall logs. The agent does both. Its permissions allowed the deletion even though the request came from someone outside the IT group.

Same agent. Same access. One added instruction. A completely different blast radius.

This is why permissions alone can't tell you what an agent will do. A tool it's allowed to use can still be used for the wrong reason at the wrong moment.

• Multi-agent systems raise the stakes. When agents call other agents, the failure surface grows.

One analysis catalogued 14 distinct failure modes across popular multi-agent frameworks and found many trace back to system design, not the underlying model.¹

Meaning a stronger model won't fix them. And they're hard to see until the whole system runs.

• When an agent acts, it's often hard to reconstruct why.

Agentic architecture can obscure what caused a given action. That makes the behavior hard to audit even after the fact.

So unpredictable behavior isn't a rare glitch. It's a property of how agents work. Careful Adoption of Agentic AI Services — guidance from the Five Eyes cybersecurity agencies — puts it plainly: assume agentic AI may behave unexpectedly. And plan for that.

The catch is where that behavior becomes visible. Not in review. Not in a scan. Only once the agent runs.

FYI: Unpredictability isn't limited to a single run. Many production AI agents retain context across sessions. So when an agent behaves unexpectedly, the cause may lie in information it encountered days ago, not the execution you're investigating.

Why can't teams patch faster than attackers can exploit?

Teams can't patch faster than attackers can exploit because the exploit now arrives first. The window between a vulnerability going public and its first real-world exploit has collapsed. In many cases, it's already gone.

Here's the trend.

The metric to watch is time-to-exploit.

It measures the gap between a vulnerability's disclosure and its first exploitation in the wild. In 2018, that gap averaged 63 days. By 2024, it hit negative one day. In 2025, an estimated negative seven, according to Mandiant.

Negative means exploitation happened, on average, before the patch was even released.

Why the collapse? AI.

AI has removed the skill and time barrier to writing exploits.

AI systems can produce working proof-of-concept exploit code for a published CVE in as little as 10 to 15 minutes, as reported by the Cloud Security Alliance in The Collapsing Exploit Window: AI-Speed Vulnerability Weaponization. Cost per attempt: about a dollar.

The Zero Day Clock, which tracks this trend, projects the median time-to-exploit will keep compressing toward one hour, then one minute.

Now the defender's side.

Patching isn't a switch you flip.

A fix has to clear compatibility testing against business-critical systems. It waits for change windows and approvals. Legacy systems may break when updated, or lack vendor support entirely. None of that moves at machine speed.

So the two clocks run on different scales.

Attackers work in minutes. Defenders work in weeks. A patch cycle measured in weeks can't keep pace with an exploit window measured in hours.

Important: this isn't an argument against patching. Patching still matters. It's an argument against relying on it as your primary control.

Here's the shift that follows.

When you can't patch before the exploit lands, speed of detection becomes the control you can actually influence. And detection happens in one place. Where the system runs.

Different force than the last section. Same destination. Runtime.

What do real AI security failures in production look like?

Real AI security failures look like a system doing something no one tested for. Using access no one thought to limit. And the gap only becomes visible once it runs.

Here's the most detailed public example.

In late 2025, Anthropic disrupted what it called the first reported AI-orchestrated cyber espionage campaign. A threat actor pointed instances of Claude Code at roughly 30 organizations. Then it let the AI run.

The agent did the work.

Reconnaissance. Vulnerability discovery. Exploitation. Lateral movement. Credential harvesting. Data exfiltration. Anthropic estimated the AI handled 80 to 90 percent of the tactical operations on its own at request rates no human team could match.

Look at where the failure lived.

None of this was a traditional exploit chain a scanner would flag ahead of time.

The tool was doing exactly what it was built to do. The danger was in the sequence of live actions it chose once running. You could only see the attack by watching what the agent did. Not by reviewing what it was.

This shape repeats. And it's not limited to attacks.

Consider what agentic AI connects to.

Protocols like MCP wire agents into content repositories, business tools, and development environments. That's useful. It's also a new trust boundary. And a trust boundary only proves whether it holds when traffic actually crosses it.

Consider the training environment too.

A model in training runs on real infrastructure with real network and compute access. Leave that access unconstrained, and the model can reach places the task never required. Which is, again, something you observe at runtime. Not in a design doc.

Here's the pattern under all of it.

In each case, the system had more capability than anyone had explicitly limited.

A sanctioned tool. A legitimate connection. A granted permission. The failure wasn't a missing control on paper. It was the gap between what the system could do and what anyone had constrained.

And that gap has one property that decides everything else.

It doesn't exist on paper. It exists in execution. A scan sees the configuration. A review sees the design. Neither sees the agent choose its next action — which is the only place the failure actually happens.

That's the reframe.

Most coverage stops at "the controls lagged behind the technology." True, but it misses the harder problem: you can't govern behavior you can't observe, and this behavior is only observable while it runs.

There's one more thing that gap decides.

You can't fix it before the fact, but you can catch it in the act. In the espionage case, that's exactly how it ended. The activity was caught in motion, then disrupted. The behavior gave itself away by running.

Why don't traditional security tools work for AI?

Traditional tools don't work for AI because they all do the same thing. They inspect the system before it runs. But AI's risk lives in what it does once it's running. So the tool checks one thing and the risk lives somewhere else.

Here's how that plays out, tool by tool.

That last row deserves a pause.

Filtering looks solid until someone adapts to it.

One study tuned attacks against twelve recent defenses. It got past most of them more than 90 percent of the time. And most of those defenses had originally reported near-zero success. So they held up. Until an attacker who understood them showed up.²

Two controls don't belong in that table. Not because they work completely. Because they don't fail the same way.

Least privilege genuinely helps.

It caps the blast radius. If an agent can't reach a system, a bad decision can't touch it. But there's a tradeoff. Constrain the agent enough to be safe, and you constrain what it can actually do. And it tells you nothing about what the agent did with the access it kept.

Human-in-the-loop is the strongest control on the list.

A person approving high-impact actions catches what automation misses. But it's also the first thing teams switch off. Why? It kills the speed and autonomy that made the agent worth deploying.

Now step back and look at what all of these share.

Each one produces a verdict at a single moment. Pass or fail, today.

But AI risk isn't a moment. It's continuous. A model that's safe on Tuesday meets a new input on Thursday and does something no test covered. A one-time check can't govern behavior that never stops changing.

So the honest read is this. None of these tools are useless. Prevention shrinks the attack surface. It caps damage. You should use all of it. But prevention is necessary, not sufficient. The exits are closed.

Why is runtime the missing layer in AI security?

Runtime is the missing layer in AI security because it's the only place the behavior exists. The build can't show you what an agent will do. The scan can't either. An AI system reveals what it does only when it runs.

So what does runtime give you that earlier stages can't?

Three things.

What's actually running.

Most teams can't fully account for their AI footprint. Which models are deployed. Which agents are active. What each one is allowed to touch.

Runtime is where you get a live inventory instead of an educated guess. It's also where non-human identities show up…AKA the service accounts and agent credentials that attackers increasingly go after.

What it's actually doing.

This is the part build-time can't reach. At runtime, you watch the real behavior. The tools an agent calls. The files and data it touches. The network connections it opens.

Watch for high-impact actions in particular: network egress, system resets, deletion of critical records. You can also baseline normal behavior, then flag when an agent drifts from it, or when its actions stop matching its stated intent.

In the moment.

Detection only helps if it's fast. When exploitation beats the patch, prevention isn't the control that scales. Time-to-detection is. Runtime is where you shorten it.

You catch the behavior as it happens. Not in a post-mortem. And catching it is what lets you act on it, interrupt a task mid-execution, or reverse what it did.

It's worth noting, though, that runtime does have a ceiling.

Runtime isn't a complete answer. It sees actions and consequences. It doesn't see the model's reasoning. You learn what an agent did. You don't always learn why it decided to.

So runtime doesn't replace the other layers. Prevention still caps the blast radius. Least privilege still matters. Governance still has to own the risk and answer for it. Runtime is the layer that was missing. Not the only layer that counts.

Here's what makes it the resolution.

Every failure mode in this article surfaces in the same place.

The unpredictable agent. The exploit that beat your patch cycle. The tool used for the wrong reason. None of them exist until execution. So execution is the one vantage point that sees all of them. And the one place you can still do something about them.

That's why runtime is the missing layer. Not because prevention failed. Because the truth of what an AI system does only appears where it runs.

→ Continue learning:

• What Is Runtime Security?
• What Is Cloud Detection and Response (CDR)?

How do you secure AI agents in production?

Start from an assumption, not a hope. Assume the agent will act in ways you didn't predict. Then build controls around that assumption instead of against it.

The earlier sections explain why. An agent decides at runtime. Its behavior first appears in production. So the job isn't to prove it's safe before it ships. It's to watch what it does and limit what it can reach.

There's no checklist that makes an agent safe. What follows are the practices that hold up in production. Adapt them to your environment.

• Know what's running.

Start with an inventory of live AI workloads including which agents are active, what each connects to, what each is permitted to do. The agents you don't know about are the ones you can't defend.

Pro tip: Give every agent its own identity, not a shared service account. Shared credentials mean you can't tell which agent acted, and one stolen key exposes all of them. Distinct identities turn your inventory into an audit trail.

• Baseline behavior, then watch for drift.

Establish what normal looks like for an agent. The tools it usually calls. The data it usually touches. Then flag deviations at the action layer. There's a sharper signal too: watch for gaps between what an agent says it's doing and what it actually does.

• Route detections into workflows you already run.

Agent activity shouldn't live in a separate console no one checks. Feed it into your existing detection and response process. But know the catch going in. Agent telemetry doesn't look like endpoint or network telemetry. Long reasoning chains and high log volume make the real signal hard to pull out, so plan for parsing that existing tooling wasn't built for.

Pro tip: Keep one audit log across all agents. When agents call each other, the trail fragments and a single action gets impossible to reconstruct. Unified logging lets you trace it end to end.

• Cap the blast radius.

Apply least privilege. Give each agent the minimum access its job requires. Then add human approval for actions where a mistake is expensive. The obvious ones: system resets, network egress, deletion of critical records. And keep the ability to interrupt a task mid-run or reverse it afterward.

Pro tip: Scope access to the task, then take it back. Least privilege limits how much an agent can reach, not how long. Use just-in-time credentials and revoke them when the sub-task ends. Access that's gone can't be misused.

• Map what you find to a framework you already answer to.

Tie agent risks to the categories in the Five Eyes guidance, the OWASP LLM Top 10, MITRE ATLAS, or the NIST AI RMF. Why? So each one has a clear owner. A finding attached to a framework has a home. A loose one gets ignored.

Notice the through-line.

None of these try to predict the agent's behavior in advance. They assume you can't. So they focus on the two things you can actually control. What the agent is allowed to reach. And how fast you see what it does.

→ Continue learning:

• What Is AI Workload Security?
• What Is the OWASP Top 10 for LLM Applications?

Is AI security in production getting harder?

Short answer: yes. The agents are getting more unpredictable and more numerous. The attackers are getting faster. Both trends are still accelerating, and neither reverses on its own.

Start with the attacker's clock.

The time-to-exploit trend isn't leveling off. The Zero Day Clock projects it keeps compressing from days toward one hour. And eventually toward one minute. Every step in that direction makes prevention less able to keep up and detection speed more important.

Now the workload side.

There are about to be far more agents. IDC projects 1.3 billion AI agents in use by 2028. More agents means more actions taken at runtime. It also means more agents calling other agents. And multi-agent systems fail in their own ways. Often from how they're wired together, not from any single model.

The links between agents grow too.

Protocols like MCP connect agents to more tools and data over time. Each new connection is another trust boundary. And each one only proves out at runtime.

The rules are hardening at the same time.

Standards like ISO 42001 and the NIST AI RMF now give AI risk a formal shape. Government guidance, including the Five Eyes agencies' work, sets expectations for how agents should be governed. Regulators are even compressing vulnerability-remediation deadlines in direct response to AI-driven attacks. Accountability is moving from optional to expected.

So the gap widens from both ends. Offense gets faster. The agent footprint gets larger. And the bar for governing it gets higher.

That scale is what changes runtime from an advantage into a requirement. Watching one agent execute is a tactic. Watching thousands, as they act and call each other, is the only way to govern a footprint that big. The more AI you run, the less optional runtime visibility becomes.

One last point. And it's the important one.

The Five Eyes guidance is called careful adoption. Not delayed adoption. The message isn't to wait for a safer generation of models. It's to adopt now, with controls that match how these systems actually behave.

Waiting isn't a strategy. It's just a slower way to reach the same risk.

Why AI security breaks down in production FAQs