This is one facet of a bespoke engagement. If you’re reading it in isolation, start there – the overall shape of the work and how I think about your business sits on the main page.

AI Early Pilot Design & Test

Most AI pilots fail because they were never designed to be tested properly. Hypothesis, success metric, control group, decision rule, then run. Two to four weeks. Scale or kill, with the data to defend either call.

Get an AI Flightcheck

Why this matters

MIT’s August 2025 study put the failure rate for enterprise GenAI pilots at 95%. The researchers called the gap between pilot and production “the GenAI Divide”. The pattern in the failures is consistent. A team gets excited about a model, runs a pilot for a few weeks, declares it “promising”, and then the pilot quietly never makes it into production because nobody set out what success was supposed to look like before the pilot started.

Twenty years ago I built live pilot testing for a financial services business. We ran it on website changes and product features. The discipline was simple – you don’t ship the change to everybody, you ship it to a measured slice, you compare what they do against the people who didn’t get the change, and you only roll it out properly if the difference is real. That same discipline maps directly onto AI rollouts and almost nobody applies it.

What this actually is

A two-to-four-week engagement where I design a pilot test for a specific AI deployment you are considering, run it end to end, and hand back a written verdict with the data behind it. Hypothesis. Success metric. Control group. Decision rule agreed in advance. Then we go.

The agreed decision rule is the part most teams miss. Before the pilot starts, we write down what number, by what date, says “ship it” – and what number, by what date, says “kill it”. When the data lands the call is already made. No motivated reasoning, no “let’s run it for another month”.

How a pilot is designed

1. Hypothesis

One sentence. “If we add AI document summarisation to the underwriting flow, time-to-decision drops by at least 20% with no rise in error rate.” Falsifiable, measurable, time-bounded.

2. Success metric

One primary metric, two guard-rail metrics. The primary one moves the business case. The guard-rails catch the failure modes the primary metric will miss – quality, error rate, customer satisfaction, edge-case behaviour.

3. Control group

A matched cohort that does not get the AI change. The pilot is meaningless without one. If a true control group is impossible, we either find a near-substitute (a previous time period, a peer team) or we redesign the pilot.

4. Decision rule

Written down before the pilot starts. “If primary metric improves by at least X with no guard-rail metric falling below Y, we ship. Otherwise we kill.” Signed off by the budget holder. The rule is the deal.

5. Run window

Two to four weeks usually. Long enough to clear novelty effects, short enough that the team is not living inside an unscoped pilot for three months.

What you get

A written pilot brief – hypothesis, metrics, cohort, decision rule, sign-off page
The pilot run – I configure the AI deployment for the test cohort, set up the measurement, troubleshoot anything that breaks during the window
A verdict report – what happened, against what we said would happen, with the data behind it
A scaling or kill recommendation – if the pilot worked, what the production rollout should look like; if it didn’t, why and what would be worth trying next

Who this is for

Teams that have been told to “do something with AI” and have a specific deployment in mind but have not yet committed budget. Also teams that have a stalled 2024 pilot that never got a clear yes-or-no – we can re-design it cleanly and run a proper test.

Common questions

Why can’t we just run our own pilot?

You can. Many teams do. The 95% failure rate is mostly teams who tried. The gap is rarely about technical skill – it is about the discipline of writing down the decision rule before the data arrives, and finding a real control group rather than waving the question away. That part is easy to skip and expensive to skip.

Is two to four weeks really enough?

For most operational AI deployments, yes. Long enough to clear the novelty effect (people behave differently in the first week of any change), short enough to bound the cost. The pilot is not the production rollout. It is the test that says whether the production rollout is worth doing. If the use case genuinely needs a longer window we say so up front.

What if the result is borderline?

The decision rule, agreed in advance, handles that. If the primary metric did not clear the bar, we kill or redesign. Borderline results are how teams end up running the same pilot for six months – the rule stops that happening. If the rule itself was wrong, that is a learning we capture for the next pilot.

Got a pilot in mind that needs a real test?

Tell me what you are thinking of deploying, who would be affected, and what success would look like in your business. I will be honest about whether a structured pilot is the right step or whether you should skip straight to a full build.