Your AI Agent Needs Benchmarks. But Keep the Vibes.

Everyone says "stop vibe-checking your AI agent." They're half right.

Vibes built our intuition. But when stakeholders asked "how do you know it's right?" — vibes had no answer.

Here's what did.

We run an AI data agent at Semrush: you ask a business question, it finds the data and answers with a number. Every week someone improves it — a new skill, a smarter prompt, a cleaner architecture.

And here's the trap: every change feels like an improvement. Nobody ships a change that feels worse.

But an agent is a system. A new skill can pollute the context for every other question. A "better" prompt can make answers slower and more expensive. You fixed the one question you tested — and silently broke ten you didn't.

Vibes can't catch that. But don't throw them away, because vibes and benchmarks do different jobs.

Vibes are a microscope. You watch the calls run one after another. You click, you probe, you see how the agent actually thinks. That's how you build intuition, spot the exact broken step, and form product vision. No benchmark gives you that — a benchmark just hands you a score.

Benchmarks are a map. They tell you whether the system as a whole got better, worse, or just more expensive. They are your regression insurance — and your only honest answer to "how accurate is it?"

Benchmarks are your trust. Vibes are your intuition. You need both.

Building the benchmark was the hard part. Our first attempt — asking analysts to write evals for their own domains — went nowhere. Writing a good benchmark is a skill in itself, and nobody is born with it.

Then came the insight: the ground truth already existed. It was hanging on the wall.

Stakeholders answer their questions with dashboards. Every dashboard has survived years of review — its default periods, filters, and metric definitions are the company's agreed answers. So take a dashboard, write down the question a stakeholder would ask it, and the dashboard's value is your expected answer. Use past periods, and the data barely moves — the eval stays valid for years.

We extracted 68 real business questions across ~20 domains this way. Every answer verified against a dashboard. Every rubric checks not just the number (±5%) but whether the calculation semantics match — right source, right filters, right date logic.

Now when someone asks "how can you trust the agent?", the answer is one sentence:

We checked it against the dashboards you already trust. Same numbers.

First benchmark run: only 38% of the questions passed.

That hurt. It was also the most useful thing that has happened to the project — because now every change has to prove itself.

Next we tested the obvious fix: a smarter model. Spoiler — it barely helped, and the reason surprised us. That's the next post.

How do you verify your AI agent's answers today — benchmarks, vibes, or hope?