Our private benchmark had started changing engineering decisions, which made optimizing against it tempting—and dangerous. I reviewed nine agent benchmarks to see what they actually grade: artifacts, declared outputs replayed in clean verifiers, dual binaries, hidden tests, expected insights, query results, assertions, self-judgments, and causal trajectories. The evaluator, not the prompt, turned out to be the real benchmark. Nine definitions of correct →
Agents’ Last Exam: the task can request a workflow that the artifact grader never inspects.Terminal-Bench: only the declared artifacts cross into a clean verifier.CyberGym: the patch creates an executable counterfactual.ExploitGym: both the outcome and the method must be correct.DeepSWE: the workspace disappears; the committed behavior is scored.InsightBench: expected-insight recall is easier to measure than complete analytical truth.
Our explorer sub-agent correlated with slower, worse answers, so removing it looked like a free win. The correlation was measuring hard questions, and the intervention moved the same work through a route that cost ten times more. Seven runs later, the final architecture was 22% faster, 68% cheaper, and more accurate but not because one bad component had been deleted. Where the work moved →
OpenAI began retiring the model generation our agent runs on, so the benchmark from the previous posts got its first real job. The successor cut its own SQL errors five-fold, yet the answers barely improved and the hard questions got worse. Sorting every failure by who owns the capability, the model or our context, explains why. Where the ceiling lives →
There's a button now in our Tableau dashboards: a chat panel that already knows which dashboard, tab, and filters you're looking at. That feature accidentally solved the problem from the previous post — where do you get a benchmark? Real questions, expert-verified answers, reproducible forever. Anatomy of one eval →
“So it gives me the answer—but is it correct?” That was the first question a VP asked me after a demo of our AI Analytics agent at Semrush. At first, my answer was, “Yeah, check the SQL it generates. You can see it in a couple of clicks.” “Sure, I can verify that because I know which table to query and how to avoid the pitfalls. But what about less technical people?” I didn't know how to answer that. The 200-skill mistake →
Andrey Breslav — who designed Kotlin at JetBrains and watched it become Google's default for Android — started Codespeak. The core idea: code is a noisy low-level representation of our intents. The real source of truth should be structured English that captures what the software does and why, compiled into code by LLMs. I gave it a shot on production code. Did it ship? →
Three minutes into a team demo, my agent forgot what we were building. The issue wasn't the model. Jira via MCP cost ~30K tokens in tool definitions alone—20% of a 150K window, before I typed anything. Agents that ship are agents that verify, and if checking work is expensive, it gets done less. Make verification cheap →
Anthropic open-sourced a performance engineering take-home: a simulated processor with 12 ALU slots, 6 vector slots, and 1536 words of scratch memory. Optimize a tree traversal kernel from its naive baseline. Think PyTorch vs. hand-tuned CUDA, stripped down to fundamentals. One evening got me to 4033 cycles; Claude Opus 4.5 hit ~1,363. What's your limit? →
Several years ago I was a math professor. My research was stability of dynamical systems: when does a system stay bounded — and when does it blow up, oscillate, or collapse to zero? LLMs have the same failure modes. DeepSeek measured Hyper-Connections amplifying signals by ~3000× — and their fix is beautifully "systems". Constraints as enablers →
When you ask an engineer to "update the ticket," you pay for it in focus. It breaks flow, turns real work into status-writing, and trains teams to optimize for appearances instead of outcomes. Manual reporting is the wrong mechanism for visibility — status updates should come from the source of truth: code. Monitor reality instead →
When you write "implement X," the AI still does research → planning → coding. You just don't see any of it, and you can't course-correct. Small mistakes in research compound into a half-good, half-bad result. So I split the work into stages I can review — and my understanding of the codebase went up, not down. The 3-stage workflow →
