NORTHELL
SYSTEMS OPERATIONAL Start a project →
AI & Automation

AI Proof of Concept: How to Scope a Pilot That Actually Reaches Production

Most AI pilots never ship. A step-by-step way to scope an AI proof of concept: pick the use case, write kill criteria, build evals and decide go or no-go.

X / Twitter LinkedIn
AI Proof of Concept: How to Scope a Pilot That Actually Reaches Production
TL;DR

An AI proof of concept should answer one question: is this worth building for real? Most fail because they prove the model can do a task on clean demo data, then stall on integration, cost and trust. Scope it the other way round: choose a narrow, high-volume task with a named owner, write measurable pass and kill criteria before any code, build an evaluation set from real data, run a thin production-shaped slice with real users, and end with a dated go, iterate or kill decision.

KEY TAKEAWAYS
  • Gartner forecast that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025, citing data quality, risk controls, cost and unclear value. Those four causes are the things a good POC should test first.
  • A demo shows the model can do the task once. A POC shows it can do it reliably on your data, at a cost and latency you can live with, inside your workflow.
  • Write the kill criteria before you build. If you cannot say what result would stop the project, you are not running an experiment.
  • The evaluation set built from real, messy examples is the most valuable output of a POC. It outlives the prototype and becomes the regression test for production.
  • Build a thin slice that touches the real system of record and real users, even in shadow mode. Notebook-only POCs hide most of the risk.
  • Not every use case needs an agent. Gartner says many use cases positioned as agentic do not require agentic implementations, and a simpler workflow is cheaper to prove and to run.
In This Article
  1. Why most AI pilots stall
  2. Demo, POC, pilot, MVP: be clear which one you are buying
  3. Step 1: Pick a use case that can be proven in weeks
  4. Step 2: Write the pass and kill criteria before any code
  5. Step 3: Build the evaluation set from real data first
  6. Step 4: Build a thin slice that is shaped like production
  7. Step 5: Put it in front of real users, safely
  8. Step 6: Measure the economics honestly
  9. Step 7: Run a real go, iterate or kill review
  10. A six-week plan you can adapt
  11. Four traps to avoid
  12. How Northell can help

Plenty of companies have already run an AI proof of concept. Far fewer have one running in production. The gap is rarely the model. A demo built in two weeks can look excellent, because it was built on twenty tidy examples by the person who chose them. The trouble starts when it meets real inputs, real systems and a finance team asking what it costs per transaction.

This is a guide to scoping an AI POC so that it answers the question that actually matters: should we build this for real? It is written for founders, CTOs and operations leaders who are about to fund one, and it is opinionated. The short version is that a POC should be designed to fail fast on the risks that kill AI projects later, not to produce the best possible demo.

Why most AI pilots stall

The numbers on this are not encouraging, though they deserve some care. In July 2024, Gartner predicted that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025, due to "poor data quality, inadequate risk controls, escalating costs or unclear business value." That was a forecast, not a measured outcome, but the four reasons it names are a useful checklist.

A year later, Gartner made a similar call about agents, predicting that over 40% of agentic AI projects will be canceled by the end of 2027. Its analyst Anushree Verma described most current agentic projects as "early stage experiments or proof of concepts that are mostly driven by hype and are often misapplied."

The most quoted figure comes from MIT NANDA's July 2025 report, The GenAI Divide: State of AI in Business 2025. According to Campus Technology's summary, the report found that 60% of organisations evaluated enterprise-grade AI systems, only 20% reached the pilot stage and just 5% reached production. Its sample was modest (153 survey responses, 52 interviews and a review of more than 300 public initiatives), so read it as a strong signal rather than a precise rate. Its explanation is the interesting part: the authors blame a "learning gap", systems that "do not retain feedback, adapt to context, or improve over time", and poor fit with day-to-day workflows. Not model quality.

Put those together and a pattern appears. AI projects rarely die because the model cannot do the task at all. They die because nobody checked, early enough, whether it could do the task reliably, affordably and inside the way people actually work.

Demo, POC, pilot, MVP: be clear which one you are buying

A lot of wasted budget comes from paying for one thing while expecting another. These terms get used loosely, so define them in the statement of work.

StageQuestion it answersTypical dataWho uses itEnds with
DemoCan a model do this at all?A handful of chosen examplesThe builder, in a meetingInterest, or not
Proof of conceptCan it do this reliably on our data, at an acceptable cost and risk?Hundreds of real, messy examplesThe team, plus reviewers grading outputA go, iterate or kill decision
PilotDo real users get value from it inside their workflow?Live data, limited scopeA small group of real usersAdoption and outcome metrics
MVP / productionCan we support this for everyone who needs it?All live dataThe full user groupAn owned, monitored product

Our view: a good POC borrows a little from the pilot stage. It should put output in front of a few real users, even in shadow mode, because user trust is one of the things that most often breaks later. A POC that never leaves a notebook has tested the easy part.

Step 1: Pick a use case that can be proven in weeks

The best first POC is not the most strategic idea on the whiteboard. It is the one where success is easy to measure and the path to production is short. We use a simple scorecard. Score each candidate and drop anything with more than one red.

CriterionGreenRed
VolumeThe task happens hundreds or thousands of times a monthA few times a quarter
Ground truthPast examples with known correct answers existNobody agrees what a good answer looks like
OwnerA named person runs the process and wants it changedSponsored by an innovation team with no process owner
Cost of an errorMistakes are caught by a human or are cheap to reverseMistakes are silent, regulated or customer-facing with no review
Data accessInputs are reachable through an API or export this monthData sits in a system nobody can query without a six-month project
BaselineYou know today's time, cost or error rateNo current measurement exists

Document triage, first-draft responses for a support or operations team, data extraction from semi-structured forms and internal search over policy documents tend to score well. "An AI assistant for the whole company" does not, because nobody owns it and nobody can say when it is good enough.

One more filter: do you need an agent? Gartner's release makes the point bluntly: "Many use cases positioned as agentic today don't require agentic implementations." If the steps are known in advance, a fixed workflow with one or two model calls is cheaper to build, easier to test and easier to explain to auditors. We cover that trade-off in more detail in single agent vs multi-agent systems.

RELATED SERVICE

Working on something like this? See our Claude Agent Development →

Step 2: Write the pass and kill criteria before any code

This is the step most teams skip, and it is the one that turns a POC from a showcase into an experiment. Before anyone writes a prompt, agree in writing what result would justify moving forward and what result would stop the project.

Anthropic's guidance on defining success criteria for LLM applications is a good template even if you use another provider. It asks for criteria that are specific, measurable, achievable and relevant, and its worked example is the right level of precision: not "classify sentiment well" but an F1 score of at least 0.85 on a held-out test set. It also points out that most use cases need several criteria at once, typically task accuracy, consistency, latency, price and privacy.

For a business POC, we recommend criteria in four groups:

  • Quality: the accuracy or acceptance rate needed on the evaluation set, plus a separate ceiling for serious errors. "92% of extractions correct, and zero cases where a wrong policy number is filled in confidently" is far more useful than one average.
  • Economics: a maximum cost per completed task, including model calls, retrieval and human review time, compared with today's cost.
  • Speed: a latency ceiling that fits the workflow. A back-office batch job and a live chat have very different limits.
  • Adoption: what share of AI output the real users accept or lightly edit during the shadow period.

Then write the kill criteria as the mirror image. For example: if accuracy on the held-out set is below 80% after two prompt and retrieval iterations, or if cost per task is above the current manual cost, we stop. Having this in writing protects everyone, including the vendor, from the slow drift where a mediocre result gets one more month, then another.

Step 3: Build the evaluation set from real data first

Collect the test data before you build the system. Pull a few hundred real past cases, with the inputs as they actually arrive (scanned, misspelt, half-complete) and the answer a competent person gave. Deliberately include the awkward ones: the edge cases your team already complains about.

Anthropic's guidance favours volume, recommending more test questions with automated grading over fewer hand-graded ones, and it ranks code-based grading as the fastest and most reliable option where the task allows it. In practice, an exact-match or field-level check on structured output can run in minutes on every change. Reserve human grading for the subjective parts, and use a model as a grader only after you have checked it agrees with your human reviewers.

Split the set. Use one part to develop prompts and retrieval, and keep a held-out part you only score against at checkpoints. Otherwise the team will tune, without meaning to, towards the very examples it reports on.

This evaluation set is the most valuable thing a POC produces. If the project goes ahead, it becomes the regression suite that tells you whether a new model version or prompt change made things better or worse. If the project is killed, it is still the evidence for why.

Step 4: Build a thin slice that is shaped like production

The prototype should be small but real. That means it reads from the actual system of record, through the same authentication and permissions production will need, and writes its output somewhere users already work: a ticket, a CRM field, a review queue. It does not need a polished UI, scaling or every edge case. It does need the same data path.

The reason is the MIT finding about workflow fit. Integration problems, permission models and messy source data are where pilots quietly die, and none of them show up in a notebook. A production-shaped slice surfaces them in week three, when they are cheap to discuss, rather than in month six.

Instrument it from the start. Log every input, output, model version, prompt version, token count and latency. You will need this to calculate economics in Step 6, and you will need it to answer the question "why did it do that?" when a reviewer finds a strange result.

Step 5: Put it in front of real users, safely

Run the slice in shadow mode or with a human approving every output. Users see the AI's suggestion next to their normal work and either accept, edit or reject it. Record which. This tells you more than any offline score, because it measures whether the output is useful to the person who has to stand behind it.

Design matters here more than teams expect. How a suggestion is presented, whether sources are shown, how easy it is to correct, and what happens when the model is unsure all change acceptance rates. We have written separately about designing UX for AI features users trust.

The shadow period also starts to address the learning gap the MIT report describes. Every edit and rejection is labelled data. Feed it back into the evaluation set and, where it makes sense, into retrieval or examples in the prompt. A POC that shows the system getting better with feedback is far more convincing than one with a single static score.

Step 6: Measure the economics honestly

Cost surprises are one of the four reasons in Gartner's forecast, and they are easy to avoid if you measure them during the POC rather than after it. Calculate cost per completed task, not cost per API call. Include retries, retrieval, the human review time you still need, and the share of cases that fall back to fully manual handling.

Compare that with the baseline you recorded in Step 1. Then project it to production volume, with a margin for longer inputs and growth. If the numbers only work with the most expensive model and no human review, that is a finding, not a detail. There are usually ways to bring the cost down through routing, caching and prompt design, which we cover in how to reduce LLM API costs, but do the maths before you promise anyone a saving.

Step 7: Run a real go, iterate or kill review

Put the decision meeting in the calendar on day one, and bring the criteria from Step 2. There are three honest outcomes.

ResultWhat it looks likeNext step
GoQuality, cost and adoption criteria met on the held-out set and in shadow modeFund a production build with the eval set, logging and owner already in place
IterateClose on most criteria, with failures clustered in one identifiable input type or stepNarrow the scope or fix the specific failure, one more time-boxed round, same criteria
Kill or parkErrors spread across all inputs, cost above the ceiling, or users not accepting outputStop, document why, keep the eval set to retest when models or data change

Kill is a successful outcome for a POC. You spent a few weeks to avoid spending a year. The failure mode to avoid is the fourth, unofficial option: the POC that nobody stops, which ends up used by a handful of people with no owner, no monitoring and no budget.

A six-week plan you can adapt

Timelines vary with data access and the number of systems involved, but for one well-chosen task this shape works well.

  1. Week 1: Score candidate use cases, pick one, name the business owner, record the baseline, write pass and kill criteria.
  2. Week 2: Collect and label the evaluation set, split out the held-out portion, confirm data access and security review.
  3. Week 3: Build the thin slice end to end with logging; first scored run against the development set.
  4. Week 4: Iterate on prompts, retrieval and workflow design; first held-out checkpoint.
  5. Week 5: Shadow mode with real users; capture accept, edit and reject data; measure cost per task.
  6. Week 6: Final held-out score, economics at projected volume, risk notes, then the go, iterate or kill meeting.

Coverage of the MIT report notes that mid-market companies moved from pilot to full implementation in about 90 days, against nine months or more for large enterprises. Speed on its own is not the goal, but a short, fixed time box forces the decisions that long programmes put off.

Four traps to avoid

  • Judging the POC by the demo. A great live walkthrough on chosen examples says little. Ask to see the held-out score and the failure cases.
  • Starting with the hardest version. Full autonomy, many tools and no human review is the riskiest way to begin. Prove the narrow, assisted version first and widen it with evidence.
  • Treating security and compliance as phase two. If customer or regulated data is involved, the data flow and vendor terms belong in week one. Discovering a blocker in week six wastes the whole POC.
  • Letting the vendor define success. Whoever builds the POC should not be the only one deciding whether it passed. The business owner holds the criteria.

How Northell can help

We run AI proof of concept projects built around the steps above: a scored use case, written criteria, a real evaluation set and a production-shaped slice, ending in a clear recommendation rather than just a demo. If you have an AI idea and want an honest view on whether it is worth proving, or a stalled pilot you want a second opinion on, get in touch.

Northell Team

Part of Northell's engineering and content team — the people who build production software, AI systems, and fintech infrastructure, and write about what actually works.

Frequently Asked Questions

How long should an AI proof of concept take?

For a single, well-defined task, four to eight weeks is a sensible time box. Less than that and you rarely get past clean demo data or reach real users. Much more and the POC turns into an unplanned build without the engineering discipline of one. The time box matters more than the exact number: agree it up front, with a decision meeting on the calendar for the last week.

What is the difference between an AI POC, a pilot and an MVP?

A proof of concept tests feasibility: can a model do this task to the required standard on your data, at an acceptable cost? A pilot tests adoption: does a small group of real users get value from it inside their workflow? An MVP is the first version you support in production for a wider audience. Many teams blur the first two, which is fine, as long as the POC still ends with a clear go or no-go rather than drifting into an unsupported production tool.

How many test examples do we need to evaluate an AI POC?

Enough to cover the variety of real inputs, including the awkward ones. In practice that usually means a few hundred real examples rather than a dozen hand-picked ones, with edge cases deliberately included. Anthropic's evaluation guidance recommends favouring volume, with automated grading where possible, over a small number of carefully hand-graded cases. Hold some examples back so you are not tuning prompts against the same set you report on.

Who should own an AI proof of concept?

A business owner who runs the process being changed, paired with a technical lead. The business owner defines what good looks like, provides real examples and decides whether the result is worth scaling. Without that person, POCs tend to be judged on how impressive the demo looked rather than whether it moved a number anyone cares about.

Should we build the POC in-house or with a partner?

It depends on whether you have people who have taken LLM features to production before, not just people who can write prompts. If you do, build it in-house and keep the learning. If you do not, a partner can shorten the path, but only if the knowledge does not leave with them. A practical middle path is to run the POC with a partner while your own engineers sit in the team, so the evaluation set, prompts and integration knowledge stay with you.

What should we do if the POC results are borderline?

Look at where it fails, not just the average score. If errors cluster in one input type you can route around, such as handwritten documents, narrow the scope and run one more short iteration. If errors are spread across everything, or the only way to hit the target is a model that breaks your cost ceiling, kill it or park it. A borderline POC that moves to production unchanged usually becomes a support burden.

GET STARTED

Need Engineers Who Ship This, Not Slides?

Tell us what you're building. A senior engineer replies within one business day with an honest read on scope, timeline, and fit — no sales rep in between.

Get a free scoping call

BONUS Book before the end of the month and we'll include a free build-vs-buy cost model for your specific project — no obligation, yours to keep either way.

We reply within one business day. No spam, no obligation.