A Good Benchmark Mostly Fails

When we demoed our analytics agent at Semrush, a VP asked the obvious question: it gives me an answer, but is it correct? I could not answer. We had two hundred skills and knowledge pages written by people who knew the data, and no way to tell whether any of them helped. So we built a benchmark: sixty-eight real business questions, each answer checked by hand against a dashboard. The agent passed twenty-six.

That was the best news of the year, because for the first time the failure had a number. This post is the explanation I give colleagues when they ask what we are doing with benchmarks and why we keep choosing questions the agent cannot answer yet.

Better is not a task

Everyone agrees the agent should be better. That sentence gives nobody a task on Monday. A benchmark is how we say what better means, not as a slogan but as dozens of concrete questions with concrete right answers. Each one is small and ordinary. Together they describe the next version of the agent more precisely than any roadmap.

And it is only the next version. The direction a benchmark sets is never the best possible one, just a good one chosen from what we know today. The value is not in one step being right but in taking steps often. Each cycle teaches us something about the agent and something about our own data.

Three words

People mix up three words. Here is how I use them. An eval is one task plus a way to judge whether the agent did it well. A benchmark is a set of evals with a philosophy: what it pushes forward and what it deliberately ignores. A regression test protects behavior we already expect to work. Evals can do that too, but I want our development benchmark to have a different job.

Regression tests are supposed to pass. A red test is a bug to investigate. If ninety-nine percent of our development evals pass, though, the benchmark gives us little direction for what to improve next. For that job, I want a benchmark that mostly fails: each failure should be a problem we cannot solve today and intend to work on next. A low score alone is not useful; the questions have to matter to our users. Tests guard the past. Benchmarks describe the future.

What about regressions?

An eval run costs tokens, time, and attention. “We did not get worse” is useful information, but I do not want it to be the only thing we buy. Some failures are cheaper to catch with ordinary checks: did the skill files load, are the required fields present, do the tools return the expected response format? Those checks can be deterministic. Whether a new skill changes the agent’s judgment still needs behavioral evals.

Harder questions often exercise the same foundations as easier ones, but that is not guaranteed coverage. A hard revenue question might reveal a broken lookup; it might also fail for another reason and hide the regression. Keep a small set of important, previously solved cases as regression checks, and spend the development benchmark budget on the next capabilities.

Descending on a flat loss

The mental model is gradient descent: measure the distance to a goal, take a step, and measure again. It is an analogy, not a promise of convergence. A saturated benchmark offers little signal about where to go next. Once roughly eighty percent of our cases pass reliably, we start building a harder one. Eighty percent is our rule of thumb, not a universal cutoff. We retire the old set as the main development target, while keeping the regression cases that still matter.

This is a cultural choice as much as a technical one. A team that only runs regression evals can learn to fear change. A team climbing a ladder can discuss which trade-offs are worth making, while protecting the behavior users depend on. That gives focus, speed, and the satisfaction of watching a number move.

How a benchmark starts

Someone takes ownership of a knowledge area, say revenue or the signup funnel. They write eight to twenty simple, unambiguous evals where right and wrong are obvious. That first set may pass on day one, which is fine, because its job is to teach eval writing and establish a baseline. Then they add harder questions with more ambiguity and more general phrasing, while keeping the grading criteria clear. Cases already answered reliably can leave the development set; the important ones stay in the regression set. When the new set saturates, they do it again.

Each cycle moves the agent, and each cycle teaches the owner something about their own domain that they did not know they knew.

Two limits

Scores are noisy. The same question has scored zero, one, and ten on three consecutive runs. For our analytics benchmark, we run each question three times and treat small score changes cautiously. Three runs help expose variability; they do not make it disappear. When we change the questions, we also change the ruler: scores from different benchmark versions are not directly comparable.

And a score does not explain why something failed, tell you whether a specific user got a wrong answer today, or replace tests. We still have to inspect the answers and traces. The benchmark tells us where to look.

In our case it turned two hundred skills written on faith into a backlog across fourteen domains: skills, table documentation, and metric definitions that need fixing. That backlog is the product.

Related posts

• Your AI Agent Needs Benchmarks. But Keep the Vibes. — Why skills and user feedback were not enough to measure DataChat’s quality.
• The Source of the Benchmark Is in Your Dashboards — Building cases from verified analytical work, and why repeated runs matter.
• The Smarter Model Wrote Better SQL. It Still Gave the Wrong Answer. — Where the ceiling of an enterprise agent actually lives.
• Dario Was Right About Alignment. It Matters. — A benchmark for the agent’s judgment, and the target of failing at least half.