By Bryan Knipe, founder of TriPinnacle Advisors
Most companies now have an AI pilot that worked. It summarized the contracts, matched the invoices or drafted the report, on a sample, in a demo, with someone watching. Then it stalled. Months later it is still a pilot, or it was quietly turned off, and nobody can say exactly why.
I have seen this from two sides. At UnitedHealth Group I ran the evaluation of AI and machine-learning investments across the enterprise, which meant looking at a very large number of AI projects at every stage, including the ones that never left the pilot. And I have built a production AI platform myself, end to end, which meant hitting every one of these problems personally. The reasons pilots stall are remarkably consistent, and so is what gets them through.
Why pilots stall
- Nobody owns the result. The data science team owns the model, the business owns the process, IT owns the systems. Everyone is responsible for a piece, and no one for whether it works in the daily work.
- "Working" was never defined. The pilot impressed people, but no one wrote down the accuracy, cost or time saving it needed to hit before the business would rely on it.
- The sample was clean. Real documents are longer, messier and stranger than the ones chosen for the demo.
- Wrong answers have no catch. One confident error reaches a customer or a manager, trust collapses, and the team goes back to doing it by hand.
- The cost was never measured. What cost pennies in the pilot costs real money at production volume, and finance finds out from the invoice.
- It doesn't fit the systems people already use. A separate tool people have to remember to open is a tool people stop opening.
1. Choose a workflow worth finishing
The best first production candidate is not the most impressive demo. It is work that is high-volume, rules-heavy, runs on documents or records, and is checked by hand today. Invoices and purchase orders, contracts, claims, reconciliations, filings and management reporting all fit. Because it is checked by hand today, you already know what an error costs and how long the work takes, which gives you a baseline to beat.
2. Agree on the numbers before anything is built
Before writing any code, I sit with the people who do the work and agree on three numbers: the accuracy the system has to reach, the time or cost it has to save, and the most it is allowed to cost per item. If those three are met on your real data, it goes to production. If not, it doesn't. Writing them down first turns a debate about whether the AI is "good enough" into a measurement.
3. Build on real data from the first week
Get a real, ugly sample of the actual work as early as possible, with the edge cases, scanned pages and exceptions included. Test against it from the first week, not the last. Most of the engineering effort in a production AI system goes into the cases the demo never saw.
4. Check every output against its source
This is the single biggest difference between a pilot and a system a finance team will trust. In the systems I build, every AI finding is checked against the document it came from before anyone acts on it, and a separate verification pass has to agree. Answers that cannot be supported are dropped rather than shown. People handle the exceptions the system flags. As the error rate proves itself, the share of items needing a person shrinks.
5. Put a cap on the cost of every run
Every model call should go through one gateway that meters it, and every run should have a hard cost limit. That makes cost per document a known number before you commit, protects you from a runaway loop, and lets you choose models per step on accuracy and price rather than defaulting to the most expensive one.
6. Fit it into the work people already do
Put the output where people already look: the system they work in, the queue they already clear, the report they already read. The AI should feel like the work got easier, not like a new application to learn.
7. Test it like software, because it is
Before anything ships, regression tests should run that deliberately try to break it: the hard documents, the known failure cases, the data a user should never see. When a model, a prompt or a vendor changes, the same tests run again. Without them, the system drifts away from what was agreed and nobody notices until something goes wrong.
8. Give one person ownership of the result and the build
This is what the forward-deployed role exists to fix. One person who understands the business problem well enough to define "working," and the technology well enough to build it, sits with the team, builds the system and runs it in the work until the numbers are met. That is how I work: the business case, the design and the build come from the same person, so nothing falls between them.
9. Hand it over properly
Production means your team can run it without me. That means documentation they can use, monitoring that tells them when accuracy or cost drifts, tests they can run, and a named owner on your side. A system that only its builder understands is still a pilot.
The short version
Pick work that is high-volume and checked by hand. Agree on the numbers first. Build on real data. Check every output. Cap the cost. Fit it into the existing work. Test it like software. Give one person ownership. That is the shape of the workflow sprint and deployment I run: two to three weeks to prove it on your data, then eight to twelve weeks to put it into production.