Evidence-Based AI Evaluation: What It Means and Why It Matters

9 November 2026

Evidence, not opinions.

Ask whether an AI assistant is ready to ship and the answer is usually an opinion. The demo went well. Testing looked fine. Nobody has complained yet.

None of that is evidence.

Evidence is what lets someone who wasn't in the room check the answer for themselves.

What evidence-based evaluation means

Evidence-based evaluation means every claim about a system traces back to a test: what was asked, what came back, how it was scored, and which version of the system answered. Someone else with the same test set and the same setup can re-run it and get comparable results.

It is the difference between “retrieval looks good” and a retrieval score on an agreed test set, a list of the questions where the right document never came back, and the trace for each one.

The first is a view. The second is something you can act on, argue with and re-check after the next change.

Why opinions fail on AI systems

Traditional software mostly gives the same output for the same input, so a tester's judgement holds up. AI systems break that in four ways.

• Answers vary between runs. Two testers running the same query get different answers. A manual check is a sample of one.
• Wrong answers sound right. A made-up policy clause reads as fluently as a real one. A reviewer skimming output ends up judging tone, not truth.
• Demos select for success. The questions in a demo are the ones someone already knew would work.
• Builders mark their own work. The people who built the system know what it is supposed to do, which makes them poor at noticing what it actually does.

Claims that haven't been tested don't hold up well once someone tests them. When Stanford researchers tested leading retrieval-based legal research tools, 17–33% of answers contained a hallucination (Stanford, “Hallucination-Free?”, 2024). The question mark in the title is the point: a claim was tested, and the evidence disagreed.

Opinion and evidence, side by side

What makes an AI testing methodology evidence-based

Every methodology page uses the same words. These are the properties that separate a method that produces evidence from one that produces a report.

Pass criteria agreed before the first run

Decide what passing means for each area, and for the questions that are most expensive to get wrong, before anyone sees a score. Thresholds set after the results tend to land just below them. Some bars are absolute: a single permission leak is a fail, whatever the average says.

A test set that looks like real use

Real questions from the people who will use the system, questions it should refuse, attempts to attack it, and the same question asked as different users. Domain experts write the expected answers and own them.

Each failure mode measured on its own

Did it find the right content? Did it answer only from that content? Did it invent anything? Does it hold up under attack? These are separate questions with separate causes and separate fixes. A blended score can rise while the one you care about gets worse, and a grounding problem looks like a retrieval problem if you only score the final answer.

Variation reported, not hidden

Because outputs vary, one run is an anecdote. Run the set more than once and report how much results move. Report latency as percentiles — p50, p95 and p99 — rather than a single average that hides the slow tail your users actually feel.

Every score traceable to its cases

Each number should open up into the cases behind it: the question, the retrieved context, the answer and the scoring decision. Record the model, prompt, index and test-set versions with every result. A finding you can't open and inspect is an assertion.

Scorers that have been checked

If a model grades the answers, compare its grades with human labels on your own content first, and report where they disagree. An automated judge nobody has validated is one more opinion, just a faster one.

Limits stated plainly

An evidence-based method says what it didn't test. A clean result on a scope that left out agent tool use says nothing about agent tool use. If the report doesn't say what was out of scope, you can't tell what the result covers.

Independence from the outcome

The test set and the scoring should come from someone with no stake in a pass. That doesn't make the builders' own tests worthless. It means they shouldn't be the only tests.

Why it matters

Evidence changes what you can do with a test result.

• Make a release decision you can defend. Ship, ship with mitigations, or hold — and be able to say why.
• Fix the right thing first. Failures grouped by cause send engineering to the retrieval problem instead of another round of prompt rewrites.
• Know whether a change helped. Re-run the same set after every change and compare like with like.
• Answer “how do you know?” when a customer, a board or a risk team asks.

That last one matters more than it sounds. Only 46% of people worldwide are willing to trust AI systems (KPMG & University of Melbourne, 2025, 47 countries). Telling people your system is accurate won't change that. Being able to show how you know is a better start.

Where breaklight fits

breaklight's assessment is built to produce evidence, not opinions. We agree the test set with you before work starts, then score each area on its own — retrieval quality, answer grounding, hallucination, adversarial and eval ops, plus AI agent verification where your system is an agent that plans and acts — using the same scoring rules every time, so results stay comparable. You receive the findings report, the performance metrics, the evidence bundle and a remediation roadmap. It is evidence for your release decision, not a certificate. The method is set out in the breaklight whitepaper.