How to choose AI testing tools without getting sold a dashboard

6 September 2026

Feature lists are designed to make every product look complete. Use a different sequence.

1. Name the release decision

What are you trying to approve?

Ship a prompt change. Swap the model. Change the retriever. Turn on a new tool. Move from internal to customer-facing. Let the system take an external action.

If you can't name the decision, you are shopping.

2. Write the behavioural contract

What the system must do, must not do, and may only do with confirmation. Quality, safety, access, latency, cost, escalation.

That contract is the oracle. Tools execute it. They do not invent a trustworthy one for you.

3. Build a representative dataset

Real examples. Known failures. Business-critical journeys. Ambiguous asks. Adversarial cases. SME-written rows beat a pile of synthetic fluff.

Segment by intent, risk, role, language, source type. Means hide disasters in small slices.

4. Calibrate the judges

If a model is scoring another model, treat that as part of the system under test.

Check obvious passes and fails. Short-correct vs long-wrong. Style and verbosity bias. Record judge version and prompt. Recheck after you change the judge.

Deterministic checks: make sure they are actually deterministic, and that formatting tricks can't buy a green score.

5. Connect offline and online

Offline: controlled dataset, candidate version. Online: real users, incomplete coverage, surprises.

Mature teams close the loop — production failures become rows, fixes get tested offline, the release gets watched again.

6. Test the measurement system

Can you reproduce the score? Does the evaluator see the retrieved chunks and tool calls? Can it tell retrieval failure from generation failure? Can a human override? Are results exportable? Can one critical failure block release even when the average is pretty?

If the answer is no, you bought a dashboard.

A maturity path that doesn't require a six-figure platform on day one

1. Make failures visible — inputs, outputs, retrieved evidence, tools, versions, latency, cost.
2. Small golden set — quality over volume.
3. Component evaluation — retrieval, grounding, hallucination, format, tools, safety. Separate them.
4. Regression gates on change.
5. Adversarial coverage.
6. Continuous eval on sampled production.
7. An assurance decision: what was tested, what wasn't, what failed, residual risk, ship / ship-with-controls / delay / narrow.

“The model scored 87%” is not an assurance decision.

Our view

Tools are evidence infrastructure. They are not assurance by themselves.

The hard work is the contract, the cases, separating retrieval defects from answer defects, calibrating judges, testing the evaluator, and making the release call explicit.

Pick the lightest stack that can produce trustworthy evidence for the risks in scope. Add complexity only when the architecture or the consequence level demands it.

A small, well-designed harness beats a large platform with weak cases. A trace with retrieved documents beats a polished score with no evidence. A calibrated, human-reviewed failure set beats thousands of synthetic rows that don't look like your users.

If you need a starting shortlist for experiments, see the previous post. If you need someone outside the build team to say whether the system is actually right — that's the job we do.

breaklight — the consultancy that tests AI — only AI. Every engagement produces evidence, not opinions.