A practical map of AI testing tools
6 September 2026
Caveat: Vendor features and pricing move. Check the docs before you buy. This is a map, not a ranking, and not an endorsement.
Once you know what you are testing, the market gets simpler.
| Job | What you're really buying | Examples people actually use |
|---|---|---|
| Prompt / model comparison | Experiments, versions, side-by-sides | PromptLayer, Braintrust, Weave, LangSmith |
| Code-first regression | Tests in CI, custom scorers | DeepEval, Promptfoo, OpenAI Evals (open-source repo; hosted OpenAI Evals platform is separate — read-only from 31 Oct 2026, shut down 30 Nov 2026), Inspect AI |
| Retrieval and grounding quality | Chunk-level and claim-level evidence | Ragas, TruLens, DeepEval, Phoenix, Giskard |
| Production traces + online eval | See what real users hit, feed cases back | Langfuse, LangSmith, Phoenix, MLflow, Weave, Braintrust |
| Agent behaviour | Trajectories, tool calls, arguments | Inspect AI, Promptfoo, LangSmith, Braintrust, MLflow |
| Security / red team | Injection, leakage, abuse | Promptfoo, PyRIT, garak, Giskard |
| Managed evaluators | Someone else's judges + workflows | Patronus, Arize-style platforms |
How we talk about a few of them in practice
PromptLayer — strong when the problem is prompt iteration with humans in the loop. Not a full application assurance layer. Check their current Tables workflow; older eval pages are marked deprecated for new work.
Braintrust — built around data, task, scorers. Useful when you want one loop from offline experiments into production scoring. Their own docs warn that aggregate scores hide ugly slices. Listen to that.
LangSmith — natural if you already live in LangChain / LangGraph. Great traces; still check exportability if you ever need an independent view of the evidence.
Langfuse / Phoenix — solid when you want more control over telemetry (open-source-oriented paths). Self-hosting does not remove the need for redaction, retention, and access control.
DeepEval, Ragas, TruLens — code-first or retrieval-specific. Good bones for a harness your team keeps. Metrics still need a real dataset.
Promptfoo — config-driven evals plus a serious red-team lane. Handy when you want both regression and adversarial coverage without six products on day one.
Inspect AI — research-grade tasks, solvers, scorers, agents. Overkill for a thin chatbot; useful when the trajectory is the product.
PyRIT / garak — security scanners and red-team force multipliers. They do not replace a human who understands your auth model and tool permissions.
Patronus and similar managed evaluators — fine if you know what the judge is doing and you can calibrate it. Hosting and privacy terms matter as much as the metric names.
Shortlists (start here, then stop adding)
Small team, retrieval assistant: DeepEval or Ragas + Promptfoo + Phoenix or Langfuse. Tiny golden set first. Grow it from real failures.
Already on LangChain / LangGraph: LangSmith first. Add Ragas/DeepEval for retrieval-specific checks. Add Promptfoo or PyRIT when the risk justifies it.
Enterprise, framework-neutral: Braintrust, Langfuse, MLflow, Phoenix, or Weave — pick on data residency, self-host, export, and whether you can capture real traces, not only the last model call.
Security-led: Promptfoo, PyRIT, garak, or Giskard — plus manual red teaming, identity tests, and tool-permission review. A scanner is not a security programme.
Agents that can do damage: trajectory capture and tool-call assertions first. Pretty answer scores second.
Higher consequence: layered stack, versioned datasets, human review, auditable release decision. Do not outsource “acceptable risk” to a vendor default.
One primary workflow. Specialist tools only where the risk pays for them. A small harness with good cases beats a large platform with thin ones.
Next: how to choose without getting fooled by feature lists — and what a real assurance decision looks like.
breaklight runs independent assessments. We will use (or sit beside) tools like these; we do not sell them. Evidence over dashboards.
Talk to us