AI Testing Benchmarks: How to Choose the Right Ones

9 November 2026

Your test set decides the release.

A new model tops a leaderboard. Your team switches to it.

Then the support queue fills with answers quoting last year's price list.

The benchmark was right about the model. It just wasn't measuring your system.

What a benchmark measures — and what it doesn't

A benchmark is a fixed set of tasks with a scoring rule. Run a model on it, get a number, compare the numbers.

That number describes one model, on one kind of task, with one prompt format, scored one way. Your system is more than the model. It is the model plus your prompt, your retrieval, your corpus, your permissions and your users' questions. A public benchmark sees none of those.

Take hallucination. The Vectara Hallucination Leaderboard puts the hallucination rate across leading models at 1.8–24.2% when summarising a single document (as of Oct 2026). That spread is worth knowing when you shortlist models. It does not tell you how often your HR assistant invents a policy clause, because your assistant isn't summarising one document it was handed. It is finding the right documents in your corpus, then answering a question from them.

Public benchmarks and your own test set do different jobs

You need both. Public benchmarks narrow the choice. Your own test set makes the decision.

Why public benchmark scores mislead

Contamination

Benchmark questions are published. Published text ends up in training data. A model that has seen the answers can score well without being able to do the task, and from the outside you usually can't tell.

Saturation

Once the leading models all score near the top of a benchmark, the gaps between them say more about noise than about capability. A benchmark everyone passes has stopped telling you anything.

Task mismatch

Answering multiple-choice general-knowledge questions is not the same job as answering from your documents. Most of what breaks a retrieval-based system — the wrong chunk retrieved, a permission ignored, a confident answer to a question your corpus can't answer — isn't in a general benchmark at all.

Setup sensitivity

Scores move with the prompt wording, the number of worked examples, the answer format and the scoring method. Two leaderboards can rank the same models differently. If you can't reproduce the setup, you can't compare the numbers.

One number

An average hides the slice that matters. A model can improve overall and get worse on exactly the questions that cost you most when they're wrong.

How to choose a public benchmark worth reading

Before a benchmark score influences a decision, ask:

• Does its task look like yours? Summarisation, question answering over documents, tool use and code generation are different jobs. Match the job, not the headline.
• Does it measure the part you're changing? If you are choosing an embedding model, a general reasoning score tells you nothing. Look at retrieval benchmarks instead.
• Is the scoring method published, and could you re-run it? A score you can't reproduce is a claim.
• How old is the question set? The longer it has been public, the more likely it has leaked into training data. Sets that keep some questions private are harder to game.
• Does it report results by category, not just one average? The breakdown is where the useful information is.
• Who ran it? Scores published by a model's own developer are marketing until someone else reproduces them.

If a benchmark fails most of these, read its ranking as a press release.

Build the benchmark that decides your release

The benchmark that matters most is the one you write. Build it from:

• Real questions from users, support logs or the people who will use the system.
• Hard cases: ambiguous wording, multi-part questions, questions that need two documents to answer.
• Questions your corpus can't answer. The right result is a refusal, not a guess.
• Adversarial inputs: instructions hidden inside documents, and attempts to pull content the user shouldn't see.
• Permission checks per role: the same question asked as different users, with different expected results.
• Critical slices, tagged: refunds, dosage, contract terms — whatever is most expensive to get wrong.

Domain experts write the expected answers and own them. An engineer guessing what correct means is the start of a test set that agrees with the system.

Score each stage separately

For retrieval, measure recall, precision, MRR and NDCG: did the right content come back, and how high did it rank? For the answer, check whether each claim is supported by the retrieved context. Score only the final sentence and you will misdiagnose retrieval failures as hallucinations.

Calibrate any model you use as a judge

If a model scores your answers, check it against human labels on your own content first, and keep checking where the two disagree. An uncalibrated judge is one more benchmark you haven't validated.

Keep your benchmark honest

• Version the test set alongside the code, and change it through review.
• Hold back a portion nobody tunes against. If prompts are adjusted until the test set passes, the test set has become training data — the contamination problem, made in-house.
• Turn every bad production answer into a new test case.
• Refresh it when the corpus changes. New documents bring new questions and new permission rules.
• Record the model, prompt, index and test-set versions with every result, so scores stay comparable over time.

Signs you're relying on the wrong benchmark

• You chose your model from a leaderboard and haven't tested it on your own questions.
• Your test set has no questions the system should refuse.
• You report one overall score.
• The test set was written by the people who built the system, after they built it.
• Nobody can say which version of the test set produced last month's number.
• A score went up and nobody knows why.

Where breaklight fits

breaklight's assessment is built on the second kind of benchmark: an agreed test set drawn from your system and your corpus, scored area by area — retrieval quality, answer grounding, hallucination, adversarial and eval ops, plus AI agent verification where your system is an agent that plans and acts — and run the same way every time, so results stay comparable. You get performance metrics for each area, a findings report and a remediation roadmap in priority order. It is evidence for a release decision, not a certificate. The method is set out in the breaklight whitepaper.