A suitable first AI project is usually not the most impressive demo. It is a bounded workflow where useful inputs already exist, a person can judge the output, and success can be measured without changing the entire organisation at once.
This guide is a decision framework, not a promise that every workflow should use AI. Privacy, security, legal, provider, and sector-specific requirements still need their own review.
Start with friction, not technology
Write down a task that consumes time, creates avoidable delay, or makes important information difficult to use. Describe what happens today before naming a tool.
A useful workflow description answers five questions:
- Who starts the task?
- What information do they receive?
- What decision or output is required?
- Who checks or uses the result?
- What happens when the result is wrong or incomplete?
If those answers are unclear, adding a model will usually automate confusion.
Score the opportunity on four dimensions
1. Value
Estimate the current cost of the friction. That may be staff time, slow response, inconsistent classification, repeated searching, missed follow-up, or rework. Choose one baseline that can be measured before the pilot.
2. Feasibility
Check whether representative inputs and expected outputs exist. A team needs enough examples to test ordinary cases and difficult cases. If the process depends on undocumented judgement, discovery may be more valuable than automation.
3. Risk
Consider the consequence of a confident but incorrect answer. A drafting assistant with human review has a different risk profile from an automated decision about money, safety, employment, or eligibility. Higher-consequence uses require stronger controls and may be unsuitable for automation.
4. Adoption
Identify who will use the system and why it would make their work better. A technically accurate prototype can still fail if it adds steps, removes useful context, or gives staff no way to correct the output.
Prefer a bounded pilot
A pilot should test one hypothesis. For example:
If an assistant retrieves approved policy passages with visible sources, staff can answer routine internal questions faster without relying on memory.
That statement identifies the user, source boundary, intended improvement, and a safeguard. It is more testable than “build an AI chatbot.”
Define the pilot dataset, evaluation cases, reviewers, time limit, cost limit, and stop conditions before building. Include examples where the correct behaviour is to say that there is not enough information.
Decide what remains human
Human review is not a temporary inconvenience. It is part of the operating design whenever judgement, accountability, or sensitive consequences are involved.
Document:
- which outputs require approval;
- when the system must escalate;
- how a user reports a problem;
- what logs are retained;
- who reviews quality over time; and
- how the feature can be disabled safely.
Measure more than task completion
Speed alone can hide new errors. Evaluate quality, correction rate, unresolved cases, user confidence, operating cost, and any change in downstream work. Compare those results with the original baseline.
The decision after a pilot is not automatically “scale.” It may be to improve the workflow, narrow the use case, change the data, add stronger review, select a different tool, or stop.
A simple readiness checklist
An AI automation opportunity is more promising when:
- the workflow and owner are clear;
- approved source data is available;
- expected output can be evaluated;
- the consequences of error are understood;
- a person remains accountable;
- success and stop conditions are measurable; and
- the team is willing to change the process around the tool.
Use the checklist to select the next question to investigate—not to justify a predetermined product.
Work through a concrete decision worksheet
Use one sheet per candidate workflow. A short, explicit record makes competing ideas easier to compare and exposes what the team is only assuming.
| Question | What to record | Evidence to collect |
|---|---|---|
| Who owns the workflow? | One accountable role, not “the business” | Process owner confirmation |
| What happens today? | Trigger, steps, handoffs, systems, exceptions | Observation, process notes, sample records |
| Where is the friction? | Delay, repetition, error, search effort, or missed action | Baseline time, volume, correction, or backlog data |
| What could AI do? | One bounded task such as classify, retrieve, extract, or draft | Representative inputs and expected outputs |
| What must remain human? | Approval, judgement, exception handling, or sensitive action | Risk review and named reviewer |
| What information is used? | Source, owner, sensitivity, access, quality, and update frequency | Data inventory and permission record |
| How will quality be tested? | Acceptance thresholds and important failure cases | Evaluation set reviewed by subject experts |
| What makes the pilot worthwhile? | Improvement over the current baseline | Agreed measure, cost boundary, and review date |
Do not combine several workflows on one sheet. “Automate customer service” might contain knowledge retrieval, identity checks, account changes, complaint handling, and escalation. Those tasks have different data and consequences and should be evaluated separately.
Example: an internal policy assistant
Consider a hypothetical organisation where staff repeatedly ask which travel expenses are allowed. The current process is to search several documents or message the finance team.
A bounded hypothesis could be: an assistant retrieves relevant passages from the currently approved policy and links to the source, while finance remains responsible for exceptions and reimbursement decisions.
The evaluation set should include ordinary questions, ambiguous requests, conflicting document versions, an out-of-policy request, and a question for which the approved material has no answer. Reviewers should check whether the correct source was found, whether the response stayed within it, whether outdated material appeared, and whether escalation occurred when required.
The pilot would not be ready merely because the demo produced fluent answers. It would need a document owner, update procedure, permission model, exception path, logs appropriate to the sensitivity, a way to report incorrect responses, and a decision about retention and provider access.
This example does not claim that the approach will fit every organisation. It shows how narrowing the task makes value and risk testable.
Common failure modes
Starting with a chatbot rather than a job
The team chooses an interface before identifying the workflow. The result can answer general questions but does not improve an owned process. Return to the trigger, decision, user, and baseline.
Evaluating only successful examples
A demonstration uses a few clean prompts selected by the builder. Real evaluation needs representative and difficult inputs, including missing information, conflicting instructions, attempted misuse, and the expected refusal or escalation behaviour.
Treating access as permission
Documents may be technically accessible but not approved for a model, a particular provider, a broader user group, or a new purpose. Confirm authority, sensitivity, retention, and contractual restrictions before moving real information.
Hiding uncertainty from the user
A fluent response can appear more certain than its evidence. Give the user source context, appropriate limitations, and a clear path to a person. Do not make high-consequence automation depend on the user detecting a hallucination.
Ignoring operations after the pilot
Model behaviour, source material, providers, costs, and user needs change. Without monitoring, a correction path, ownership, and a disable/fallback plan, a promising prototype can become an unmanaged production risk.
Next-step checklist
Before approving a build, confirm that the team can answer yes to these questions:
- Is one workflow and one accountable owner named?
- Is there a baseline for the current problem?
- Can representative inputs be used lawfully and safely?
- Can subject experts define acceptable and unacceptable outputs?
- Are high-consequence actions kept behind suitable human or rule-based control?
- Are privacy, security, provider, retention, and access decisions recorded?
- Does the evaluation set include failure and refusal cases?
- Are pilot cost, duration, success threshold, and stop conditions agreed?
- Is there a realistic integration and adoption path if the pilot works?
- Is someone responsible for monitoring, correction, and retirement after launch?
If several answers are no, the next useful investment may be process clarification, data preparation, or policy work rather than an AI build.
Primary reference
- NIST AI Risk Management Framework and operational resources — a voluntary framework for considering trustworthiness across AI design, development, use, and evaluation.