Your AI Agent Needs Benchmarks. But Keep the Vibes.

“So it gives me the answer—but is it correct?” That was the first question a VP asked me after a demo of our AI Analytics agent at Semrush. At first, my answer was, “Yeah, check the SQL it generates. You can see it in a couple of clicks.” “Sure, I can verify that because I know which table to query and how to avoid the pitfalls. But what about less technical people?” I didn't know how to answer that.

Wrong start

Our first attempt to solve the problem was to ask employees who knew the data well to write skills that the agent could load on demand without polluting the context.

If the experts encode their knowledge, correctness takes care of itself. A good data governance strategy would force them to keep those skills good, and the problem would go away, right? Right? RRIGHT?

Unfortunately, the opposite happened. Writing a skill is an art in itself.

There are several research papers showing that adding a skill may do nothing at best—and, at worst, may confuse the agent and make it dumber.

We reached a point where we had more than 200 skills, tables, and internal knowledge pages. Together, they created a slopmold that probably didn't improve the overall accuracy of the answers, just made the ticket closed.

But we had no way to measure that “probably.”?

On the right path

The way the industry measures agent quality is through evals.

A simple eval consists of a task for your agent plus a set of criteria for determining whether it did a good job. When you combine a group of evals, you get a benchmark.

So I created around 30 evals based on questions users had asked, and I asked my team to do the same.

As it turned out, creating a good eval is hard.

Nobody is good at it at first, because evals look like tests but aren't. We treated them like unit tests—something that should pass 100% of the time.

But a benchmark is a ladder you climb. A good one exposes weaknesses, then you fix the context, then evals saturate, and then you build a new, harder benchmark.

We didn't know that yet. We spent an entire day on a single question, trying to determine the right way to interpret “revenue last quarter” or “signup flow” in that particular context. On top of that, you need good examples of evals—and we didn't have any. So our evals came out so ambiguous that we couldn't trust their own verdicts.

Got derailed on vibes

Which left us right back where we started: vibes. Ask a question, examine the sequence of tool calls, inspect the output. After 30 queries, I started to get a feel for what's wrong, then I fixed the problem and ran a few queries again. Vibe-checking was embarrassingly effective—and an ideal way to bus-factor yourself to burnout. At some point the hurdle overwhelmed me and I transitioned to hoping that maybe this time it would just work…

Another hope was to evaluate the agent online—the famous thumbs-up and thumbs-down buttons.

We had a few hundred users, and only a handful of them ever clicked.

And even then, if you're not an expert, how would you know whether the answer is correct? The agent that flatters you gets a thumbs up; the agent that asks “Which cohort are you interested in?”, “Last 30 days or 90 days?” gets a thumbs “FYOU! If I knew, I could do that myself.” Thumbs measure satisfaction and reward sycophancy.

Big pause

So: skills don't measure accuracy. Thumbs are garbage. Vibe-checking doesn't scale. And creating evals from users' questions is so hard it probably requires a separate skill set. We even pitched hiring a dedicated eval engineer. That headcount was pushed to the next quarter…

But then I discovered a way to create our first benchmark—one that was verifiable, reliable, and built in days, not quarters. The source wasn't users' questions at all. It had been hiding right in front of me the whole time. And you can definitely do it too.

In the next post, I'll show you how I did it, what an eval looks like in practice, how to automate it, and what broke along the way.