Define the task before the technology
“We should use AI” is a direction, not a testable brief. Pick recurring work with a known starting point and an output someone can assess. Finding relevant information in a defined document collection is a more manageable first scope than handling all company knowledge. Name the task owner and users, and record how they do the work today.
Assess value and data readiness together
Frequency alone does not make a task suitable. Check whether the information is current, which source takes precedence and who may access it. Conflicting documents can undermine an otherwise useful scenario. If preparing the data is part of the work, include it in the plan with a named owner instead of treating it as an invisible prerequisite.
Consider what happens when an answer is wrong
The controls for an information suggestion should differ from those for an action that is hard to undo. Discuss the consequences of incomplete or incorrect output. Decide who reviews the result, when a person must approve a step and when the system should stop or decline. These boundaries belong in the evaluation criteria.
Evaluate real work
A convincing demonstration can depend on unusually favourable examples. Build an evaluation set from representative tasks, with incomplete information, conflicting sources and requests outside the agreed scope. Define useful behaviour in advance. Relevance, source quality, permissions and a clear expression of uncertainty may matter as much as whether the answer sounds right.
Compare against the current method fairly
Measure the existing task before judging the pilot. Account for finding information, completing the work and checking the result. Apply the same view to AI-assisted work: generation time alone misses review and correction. Compare like-for-like tasks so a fast answer does not hide a slower overall process.
End the pilot with a decision
Before starting, agree what evidence would support expansion, revision or stopping. A pilot might demonstrate a useful application, or reveal that the data needs attention first. Either can inform a sound decision. Close the exercise with findings, limitations and unresolved questions rather than letting the pilot continue without a purpose.