A pilot proves that a model can produce a useful answer on a curated sample. Production asks a harder question: can the same behaviour survive messy inputs, changing policy, staff turnover and an auditor who wants to know why a decision was made. The gap between those two questions is where most enterprise AI programmes quietly end.
We treat the move to production as an engineering and governance exercise rather than a bigger pilot. That means defining the decision the system supports, the evidence it must cite, the cases it must refuse, and the human who owns the outcome — before any scaling work begins.
Start from the decision, not the model
Write down the decision the system participates in, who is accountable for it today, and what a good and bad outcome look like in business terms. If that sentence cannot be written, the pilot is a demo rather than a candidate for production.
- Name the decision owner and the review path for disputed outputs
- Define the input set the system is allowed to see, and the retention rules for it
- Agree what happens when confidence is low: refuse, escalate or route to a queue
Build the evaluation set before scaling
An evaluation set curated with the people who do the work today is the single most useful artefact in an AI programme. It converts opinion into a repeatable check, and it makes model or prompt changes safe to ship.
- Sample real historic cases, including the awkward ones nobody likes
- Have senior practitioners label expected behaviour, not just expected text
- Re-run the set on every model, prompt, retrieval or policy change
Make access control part of retrieval
Retrieval that ignores permissions is a data incident waiting to happen. Role-aware retrieval — where the index respects the same entitlements as the source systems — is far easier to build early than to retrofit after go-live.
Design the exception path first
Automating the happy path is straightforward. The operational value shows up when exceptions have owners, SLA timers and a visible queue, so the organisation can see what the system could not handle and why.
Instrument benefit, not activity
Usage dashboards flatter a programme. Cycle time, rework rate, exception volume and cost per case tell you whether the work changed. Agree those measures with finance before launch so the results are not contested later.