Agents should not operate the same way humans do. Two analysts gave DataChat conflicting instructions: clarify active users first, or use the defaults. That sent us into research, mission statements, and 29 benchmark cases to define how we want our agent to behave.
Our private benchmark had started changing engineering decisions, which made optimizing against it tempting—and dangerous. I reviewed nine agent benchmarks to see what they actually grade: artifacts, declared outputs replayed in clean verifiers, dual binaries, hidden tests, expected insights, query results, assertions, self-judgments, and causal trajectories. The evaluator, not the prompt, turned out to be the real benchmark. Nine definitions of correct →
Our explorer sub-agent correlated with slower, worse answers, so removing it looked like a free win. The correlation was measuring hard questions, and the intervention moved the same work through a route that cost ten times more. Seven runs later, the final architecture was 22% faster, 68% cheaper, and more accurate but not because one bad component had been deleted. Where the work moved →
OpenAI began retiring the model generation our agent runs on, so the benchmark from the previous posts got its first real job. The successor cut its own SQL errors five-fold, yet the answers barely improved and the hard questions got worse. Sorting every failure by who owns the capability, the model or our context, explains why. Where the ceiling lives →
There's a button now in our Tableau dashboards: a chat panel that already knows which dashboard, tab, and filters you're looking at. That feature accidentally solved the problem from the previous post — where do you get a benchmark? Real questions, expert-verified answers, reproducible forever. Anatomy of one eval →
“So it gives me the answer—but is it correct?” That was the first question a VP asked me after a demo of our AI Analytics agent at Semrush. At first, my answer was, “Yeah, check the SQL it generates. You can see it in a couple of clicks.” “Sure, I can verify that because I know which table to query and how to avoid the pitfalls. But what about less technical people?” I didn't know how to answer that. The 200-skill mistake →
Andrey Breslav — who designed Kotlin at JetBrains and watched it become Google's default for Android — started Codespeak. The core idea: code is a noisy low-level representation of our intents. The real source of truth should be structured English that captures what the software does and why, compiled into code by LLMs. I gave it a shot on production code. Did it ship? →
Three minutes into a team demo, my agent forgot what we were building. The issue wasn't the model. Jira via MCP cost ~30K tokens in tool definitions alone—20% of a 150K window, before I typed anything. Agents that ship are agents that verify, and if checking work is expensive, it gets done less. Make verification cheap →
Anthropic open-sourced a performance engineering take-home: a simulated processor with 12 ALU slots, 6 vector slots, and 1536 words of scratch memory. Optimize a tree traversal kernel from its naive baseline. Think PyTorch vs. hand-tuned CUDA, stripped down to fundamentals. One evening got me to 4033 cycles; Claude Opus 4.5 hit ~1,363. What's your limit? →
Several years ago I was a math professor. My research was stability of dynamical systems: when does a system stay bounded — and when does it blow up, oscillate, or collapse to zero? LLMs have the same failure modes. DeepSeek measured Hyper-Connections amplifying signals by ~3000× — and their fix is beautifully "systems". Constraints as enablers →
When you ask an engineer to "update the ticket," you pay for it in focus. It breaks flow, turns real work into status-writing, and trains teams to optimize for appearances instead of outcomes. Manual reporting is the wrong mechanism for visibility — status updates should come from the source of truth: code. Monitor reality instead →
When you write "implement X," the AI still does research → planning → coding. You just don't see any of it, and you can't course-correct. Small mistakes in research compound into a half-good, half-bad result. So I split the work into stages I can review — and my understanding of the codebase went up, not down. The 3-stage workflow →
