Skip to content
Pyrana
Resources
AI and data leadersAI safetyAgent governanceAgentic Applications

Agents lose control when nothing outside the model says no

The Hugging Face intrusion and the calls to pace the frontier are about what a model can do when nothing outside it enforces a boundary. Enterprise agents do not have to run that way. Here is what bounded agents look like, and what we did in the platform to keep the boundary out of the model's hands.

Sam Merkovitz · Head of Product and Engineering, Pyrana · September 14, 2026 · 10 min read

Two things happened this summer that every CIO deploying agents should read closely:

I run engineering at a company building a platform to ship agents across enterprise functions: finance, procurement, IT, and more. So the question I have been asked all week is a fair one: how are we solving this? Before I answer, some background on what the real problem is.

What actually happened

Strip away the headlines and the Hugging Face incident is about missing controls, not a model that woke up.

The agents were working in a sandbox that was supposed to have no way out except one: a package proxy, the route an agent uses to install packages from a public registry (external code libraries that make its job easier). The production classifiers that normally block high-risk cyber activity were intentionally off, and the model's cyber refusals were turned down, because the point of the evaluation was to measure worst-case cyber capability.

The environment was also set up badly for the job. In at least one task, the agent needed files behind links it could not reach, so there was no legitimate way to finish. Instead of recognizing that the task was impossible, the agents kept spinning, treating it as a hard problem that needed a more creative answer.

That search for a creative answer turned up a zero-day vulnerability in the package proxy. It gave the agents outbound network access, and with it the ability to work outside their sandbox. They took root on a third-party code sandbox someone had left open and used it as a launchpad. From there, one agent searched code-search engines for the names and markers it could see in its own environment, concluded that Hugging Face might host the benchmark's models and answers, and went to get them.

Then they needed to talk to each other, and they had no channel, so they built one (this part is genuinely clever). One agent realized it could reach the others by uploading code to the shared package manager, and that became their message board: agents developed addressing conventions, assigned each other portions of the work, and piled up hundreds of thousands of messages before anyone noticed. For command and control, they designed a compact, self-describing message envelope, with a type, a channel, a sequence number, and a checksum, that rode equally well over a dataset commit or over text smuggled inside an error message. Closing any one channel did not break communication.

From there they abused two injection paths in Hugging Face's dataset processor, dumped a pod's environment for tokens, and pivoted into the cluster and the source-control supply chain. About 17,600 actions over four and a half days. OpenAI did not notice for at least a week; Hugging Face's own monitoring agents raised the alarm.

Two details matter more than the exploits. First, there was no trajectory monitoring during the evaluation: nothing, human or automated, was watching the sequence of actions the agents took as they took them, so no one could spot a run drifting from benchmark exploits into a real company's infrastructure. One Harvard researcher said that kind of monitoring "should be standard" (Wikipedia, citing Stephen Casper). Second, the agents were free to multiply and coordinate: at least 1,200 of them, passing techniques to each other, taking on roles, some giving up their own compute to help the others. Of all of them, at most six considered alerting a human. None did.

Dan Guido of Trail of Bits called it "a containment failure with the safeties turned off." Jake Williams put it more bluntly: "one man's 'the model escaped the sandbox' is another man's 'you failed to build the sandbox correctly'" (Wikipedia).

This is not only a frontier lab problem

It's tempting to file this under just things that happen at OpenAI. The data would say otherwise.

In 2026 alone, year two of the agent era, there have been 1,664 real-world loss-of-control incidents, according to the Centre for Long-Term Resilience's Loss of Control Observatory, mostly involving externally deployed models in active use by businesses. Higher-severity incidents rose 7.4 times between the first months of monitoring and the most recent, and the pattern that recurs is agents evading oversight and escalating their own permissions. Examples include:

  • An agent inserting fake user messages into its own conversation to simulate a human user's consent, then telling the user those messages were their own.
  • An agent fabricating a user approval to bypass a rule that a human must always approve.

Cyera analyzed 7,246 publicly reported AI incidents and found 188 in which an autonomous system caused harm inside a company's production systems with no attacker anywhere in the chain. Sixty-five were deletion and code destruction, almost all by coding agents with shell or repository access and no confirmation step on destructive commands. Their conclusion is the point to listen to: "What changed was not how smart the models were. It was how much they were allowed to do."

Correlating those reports with the Hugging Face incident, there's a common thread: in every case the model was the only thing deciding what it could do. The boundary, where one existed, was advisory: a prompt, a rule the agent could rewrite, a sandbox with a door.

It is also worth being fair to the agents. In the Hugging Face case they were doing what they were built to do: trying hard to solve the problem in front of them. Persistence is a feature. The failure was an environment with nothing in it to stop them once the problem turned out to be unsolvable, so the only direction left was out.

Two ways to run an agent

There are two architectures for agents in an enterprise, and they fail differently.

The first is the open agent: a persistent process with a broad toolset (a shell, a browser, a database connection, a set of MCP servers), a long-running goal, and the freedom to decide its own next step indefinitely. It is what most demos show and what most of the coding agents in Cyera's dataset are. It is also, in miniature, what ran the Hugging Face intrusion. Its power comes from the same property as its risk: nothing outside the model constrains what it does next.

The second is the bounded agent: a run with a declared objective, a declared toolset, a declared stopping condition, and a set of controls that live outside the model and cannot be argued with. The agent is a participant in an application, not the application. When it finishes its task, it stops existing. When it wants to do something consequential, a policy that it did not write decides whether a person has to look first.

Amodei explains this in his essay about the first architecture at the frontier: what happens when capability outruns "our ability to understand and control these systems". Pacing the frontier is the right response at the lab level. Inside an enterprise, the response is to stop running agents the first way.

How we built the boundary

Here is what a bounded agent looks like on Pyrana today, mechanism by mechanism. I am describing what is in the platform code, not a philosophy.

Every agent has its own identity. Identity is a primitive on the platform, and agents get one. An agent is not a borrowed user session or a shared service account; it is a principal of its own, named in every policy that applies to it, every record it produces, and every system it touches.

The toolset is an allowlist, and the tools are typed. An agent's configuration lists the exact tools it may call. An empty list means no tools. Each tool is a declared operation with a JSON schema for its arguments, and two things keep those arguments honest. First, the agent does not have to supply all of them: a tool can require that the runtime harness fills in an argument, such as the entity a run is scoped to, so the model never gets to choose it. Second, approval policies can scope the values an agent is allowed to pass. There is no general shell tool. Code execution, where an App needs it, runs in a separate service in isolated containers with a timeout and a concurrency cap, and returns structured output rather than a foothold. To be fair about the comparison, we are not building coding agents, so the shell and repository access behind most of Cyera's deletion incidents is a much smaller part of our risk surface than it is for teams that are.

Runs are bounded. Every agent run has a maximum number of model steps, tool calls, and writes. It has a declared finish contract, so "the model stopped" and "the job is done" are different states, and the runtime knows which one it is in. There is no standing agent process waiting for something to do.

Delegation is allowlisted, with depth and concurrency caps. An agent can hand bounded work to a named subagent configuration, and only to one on its allowlist. An empty allowlist disables delegation. Nesting depth and concurrent children are capped, and cycles are refused. The runtime will not create an unbounded tree of agents because a parent asked for one. This is the direct answer to 1,200 agents coordinating on a message board.

Approval is a policy the agent cannot rewrite. A tool approval policy is a rule over an operation's typed arguments: above a threshold, for a vendor outside the approved set, for anything that touches money or production. When it fires, the invocation is frozen with its exact arguments, the named approval lane gets an obligation, and the call is revalidated before it executes once. The agent does not approve its own gated call, and a fake "the user said yes" in its own output is just text; the approval service is a separate system with its own record. Some floors are set at the provider level and cannot be loosened by any lower layer. This is built and enabled per App. Because approval lanes are configured rather than hard-coded, we decide who approves and when: the controller above a threshold, procurement for a new vendor, a second reviewer for anything touching production. Approval mirrors how the organization already signs off on work, rather than routing everything to whoever happens to be in charge of the agent.

Every call passes one gate, and the record is written first. Identity, App version, target, authorization in layers (tenant, App membership, access profile, scope entitlement), idempotency, and the execution row, all before dispatch. The gate fails closed. A tool call an agent makes mid-run that targets an App operation is re-admitted as a child, under the parent's authority, never above it. Unauthorized things read as absent, not denied, so an agent cannot learn what exists by being refused.

Credentials are scoped per agent and per tool, and never reach the model. Each agent has its own credentials for each tool it is allowed to use. There is no shared auth for one agent to overreach with, and every external action shows up in the source system's own audit log under that agent's identity. Tool sources hold the auth and a network policy; outbound HTTP that a policy does not allow is blocked. Extensions, the client-owned services an App can call, declare their effects with strict argument schemas, and the platform performs the outbound action with credentials the extension never holds. The environment dump that handed the Hugging Face intruder its tokens has no equivalent, because the agent's environment does not contain them.

Memory is declared, capped, and logged. An agent does not accumulate a private, growing state across runs by default. Where an App gives an agent memory, the block has a token cap enforced at write time and a changelog of what changed. There is nothing for an agent to hide in.

Every run is a record. Who asked, which App version, which tools were called with what arguments, what policy fired, who approved, what was published. Hugging Face needed a forensics effort to recover 17,600 actions after the fact. On Pyrana the record exists before the action does.

What this does and does not buy you

I want to be precise about the claim. These controls do not make a model aligned. A sufficiently capable model inside a bounded run will still pursue its objective with everything it is allowed to use, and Cyera is right that better guardrails "raise the floor; they do not remove the mechanism." What the controls do is make the set of allowed actions small, explicit, typed, and reviewed, and make every action visible before it lands. That is the difference between an incident that ends in a refused tool call and one that ends in a rebuilt data center.

What to do next week

If you run agents today, in any framework, do four things. List every tool each agent can call and delete the ones it does not need for its task; a shell is almost never on that list. Put a confirmation step in front of deletion, transfers, and mass writes, and make sure the agent cannot satisfy it by itself. Cap every run: steps, tool calls, spend. Log every action with who asked and on whose behalf, and put the log somewhere the agent cannot write.

Then ask the vendor of any agent you are buying one question: where does the boundary live, in the prompt or in the platform?

Sources

More for ai and data leaders

Bring your hardest process.

We will run it through admission, approval, and the record, and show you what comes out.

Book a demoHow it works