Skip to content
Pyrana
For business leaders
AI pilots
Production readiness

My AI pilots keep failing. Now what?

Most AI pilots are scoped so the demo works, which is why they die in the security review. A production-first pilot answers identity, approval, and audit on day one, and the five questions to ask before you fund the next one.

Eric Tarnowski
September 9, 2026
7 min read

The third pilot

You have run two or three of them. Each one demoed well. One is still in a sandbox 18 months later. One got as far as a security review and stopped. One went live for a team of six and quietly stopped being used. The vendor has moved on, the slide deck is still in the board folder, and someone has asked what happened to the budget.

I have sat on both sides of that meeting. Here is what I think is going on, and what I would do differently next time.

The pilot was built for the demo

A pilot is scoped so the demo works. That sounds obvious. The problem is what the scoping leaves out.

The demo shows the part above the waterline: a model, a prompt, a few tools, a chat window. It runs on a sandbox copy of the data, or on a read-only production view that someone quietly granted. It authenticates as the person who built it. The "human in the loop" is a colleague watching a Slack channel. Nothing it does has an approval threshold, because nothing it does matters yet.

Then someone tries to move it into production, and every question it was scoped to avoid arrives at once. Security asks what identity the agent runs under and what it can reach. Finance asks who approves a journal entry it proposes and where that approval is recorded. Internal audit asks for the record of what ran, on which version, with which data. The process owner asks what happens when the vendor API times out halfway through a payment run. The pilot team, which was measured on the demo, owns none of those answers.

The pilot does not fail in the sandbox. It fails in the meeting where those questions get asked, because the honest answer to each of them is "we would have to build that."

The numbers say this is the normal outcome

MIT's NANDA initiative reviewed more than 300 enterprise GenAI initiatives in 2025 and found that 95 percent of pilots produced no measurable P&L impact. The ones that worked were narrow and embedded in a workflow, and the gap was learning and integration rather than model quality (MIT NANDA, "The GenAI Divide," reported in Fortune, August 2025). McKinsey calls the same thing the gen AI paradox: nearly eight in ten companies use gen AI, about as many report no significant bottom-line impact, and roughly 90 percent of function-specific use cases are still in pilot (McKinsey, June 2025).

Gartner predicted in June 2025 that more than 40 percent of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value, and inadequate risk controls. Note the third reason. In May 2026 Gartner added a second prediction: by 2027, 40 percent of enterprises will demote or decommission autonomous agents because of governance gaps that surface only after a production incident, and it named binary governance, either locked down or fully trusted, as the root cause. Forrester's June 2026 read is the bluntest. Three quarters of enterprises say they are adopting agentic AI, few have anything in production beyond what Forrester calls "agentish" chatbots, and its summary is "Interest is everywhere. Scale is rare."

Read those together and the pattern is the one you saw in your own building. The models work. The layer under them was never built, and the pilot was designed so nobody would notice until it was time to ship.

What a production-first pilot looks like

The fix is to run the pilot as if it were already in production, on a process that is small enough to finish and painful enough to matter. Four rules.

Pick the process that hurts. It has a named owner, a cycle time you can measure today, and a cost you can put in one sentence. Month-end close for one business unit. Vendor onboarding. Collections follow-up on the 60-day bucket. "Ask questions about our documents" has no baseline to beat, so it can never tell you whether it worked.

Use real identities from day one. The people who call the agent sign in through your SSO. The agent has an identity of its own, with its own credentials in the systems it touches, and that identity can reach the finance share and cannot reach the HR share. If the pilot runs on the builder's login, you have learned nothing about production.

Put the real approval in the path, with a threshold and a record. If the controller must approve journal entries above 25,000 USD, then the pilot pauses on a 41,800 USD entry, the controller approves it somewhere that keeps a record, and the entry posts after the approval, never before. A colleague reading a channel is a demo. An approval policy with a record is a control.

Measure cycle time and exceptions. Days from period end to close. Share of items that needed a person. Rework rate. Cost per item. Agree on the kill criteria before the first run, so the decision at week eight is a comparison rather than a mood. Delight is not a metric.

Then decide who owns the rules. The threshold, the approver, and the list of systems the agent may touch belong to the controller or the procurement lead. If the rule owner cannot change a threshold without a developer, you are still in a pilot, whatever the status report says.

How we do it at Pyrana

We built Pyrana so the production questions are answered before the first run, once, and then reused by every application after it.

An Agentic App is defined as configuration: which operations it exposes, which execution lane runs each one (a direct lane with no model for deterministic steps, an agent lane, a durable workflow lane on Temporal, batch, event), who may call what on which data, and what needs approval. That configuration is compiled into an immutable, content-digested version and activated by moving one pointer. Rollback moves the pointer back. The answer to "which version ran" is always exact.

Every call into the App, whether it comes from the App's UI, from chat, from an API, from another agent over MCP, or from a schedule, passes one admission check: who is calling, under what authority, on which data, with what approval. People arrive through SSO. Agents arrive with their own identity and their own credentials. Admission fails closed, and the audit record is written before anything runs. Unauthorized items are absent from results rather than "denied," so an agent cannot learn what exists by being refused.

Approval is a policy over the actual arguments rather than a flag on a tool. We built conditional approval so that above a threshold the invocation, its arguments, and its evidence are frozen where the waiting agent cannot alter them; each named approval lane (the requesting user, a group such as finance, the admins) gets its own obligation; the reviewer's eligibility is checked at decision time; and the frozen invocation is revalidated before it dispatches exactly once. If the payment provider returns an ambiguous answer, the call goes to reconciliation instead of a blind retry. This is built and enabled per App.

Here is what that looks like in a first App. A PE-backed hospitality group runs close for 14 properties. The App's agent lane drafts accruals and reclasses from the property ledgers overnight and posts entries below 25,000 USD directly. A 41,800 USD entry freezes and goes to the controller lane. She approves it at 7:40 a.m. from the App, and the record shows the entry, the argument values, her approval, and the App version that ran. The first App is live in weeks. The identity model, the access profiles, the approval policy, and the audit record are now assets. The second App, say vendor onboarding, starts with all of them, which is why the tenth application costs less than the first and why the security review never has to be repeated from scratch.

Knowledge compounds the same way. cortIQ (formerly the Context Engine) extracts the close policy and the chart-of-accounts rules into Context Units (CxUs), each one a single claim with its verbatim supporting quotes and its provenance, and every run records what was retrieved, what was injected into the prompt, and what was cited. The vendor onboarding App reads the same units under its own permissions. Nothing is uploaded twice, and a policy change lands in every App that reads it.

Five questions to ask before the next pilot

  1. Which named process, with which measured cycle time and cost today, is this pilot meant to change?
  2. Under what identity will the agent run, and in which systems will that identity exist with its own credentials?
  3. Which actions require approval, at what threshold, by whom, and where will the approval be recorded?
  4. Show me the record of one call that was refused and one that was approved.
  5. Who owns the rules after go-live, and can they change a threshold without a developer?

If a vendor, or your own team, cannot answer all five before the first run, you are about to fund a fourth demo.

Sources

Continue the Conversation

Building production-grade agentic AI requires context, orchestration, and human-in-the-loop design. We'd love to show you how Pyrana delivers it end to end.

Request a Demo