How to Evaluate Whether an AI Solution Is Working
A practical method for measuring an AI system after launch, including what to track, how to review it and when to switch it off.
AI systems are easy to launch and hard to judge. A demo looks impressive. Three months later nobody is sure whether it is helping. Without a deliberate way of measuring, you end up relying on anecdotes, and the anecdotes people remember are the failures.
Here is a method that works for any small-business AI system.
Step 1: Define the job in one sentence
"Answer calls we would otherwise miss and book estimates." "Reply to website inquiries within one minute with accurate information." "Extract invoice details into our accounting system."
If you cannot write the sentence, you cannot evaluate the system, because you have not decided what it is for.
Step 2: Record the starting point
Before switching it on, write down the current numbers for whatever the system should improve:
- How many calls go unanswered each week?
- How long does a new inquiry wait for a reply?
- How many hours does the task take?
- How many errors do you catch?
Rough figures are fine. Without a before, there is no after.
Step 3: Track three kinds of measure
Volume. How much is it handling? Calls answered, messages processed, documents read. A system that handles five things a month is not earning its keep, however well it does them.
Quality. How often is it right? This requires a person to review a sample.
Outcome. Did the thing you cared about change? More appointments booked, faster replies, fewer hours spent.
| Measure | Example for an AI receptionist | Example for document extraction |
|---|---|---|
| Volume | Calls answered per week | Documents processed per week |
| Quality | Share of calls handled correctly | Share of fields extracted correctly |
| Outcome | Appointments booked from those calls | Hours of data entry removed |
Step 4: Review a sample every week
This is the step people skip, and it is the one that matters most.
Each week, look at a random selection of what the system did. For a phone system, listen to ten calls. For an email assistant, read twenty replies. Mark each one:
- Correct. It did what it should.
- Acceptable. Not ideal, but no harm done.
- Wrong. It gave bad information, missed the point or failed to hand over.
Pick the sample at random. If you only look at the calls somebody complained about, you will conclude the system is terrible. If you only look at the ones it flagged as successful, you will conclude it is perfect.
Step 5: Sort the failures
Each wrong answer usually has one of four causes:
- Missing information. It did not know something it should have. Fix: add it.
- Unclear instructions. It was not told how to handle this situation. Fix: update the instructions.
- Outside its job. It tried something it should have handed to a person. Fix: tighten the handoff rules.
- A genuine model error. It had what it needed and still got it wrong. Fix: sometimes none, which tells you whether the task suits AI at all.
The first three are fixable, and in a new system they account for most problems. A rising share of the fourth type is a warning sign.
Step 6: Ask the people affected
Numbers miss things. Ask staff whether it is saving them time or creating clean-up work. Ask a few customers how the experience was. One conversation can reveal a problem no dashboard shows.
When to switch it off
Be willing to conclude it is not working. Consider stopping if:
- Quality has not improved after several rounds of fixes.
- Staff spend more time correcting it than the task used to take.
- Customers are actively avoiding it.
- The outcome you cared about has not moved.
Turning off a system that does not work is a good decision, not a failure.
A realistic expectation
A well-built system tends to start out acceptable and improve noticeably over the first few weeks as gaps are found and filled. If it is not getting better with review, the review is not being done or the job was the wrong one to automate.
