Marat Yuldashev

Dario Was Right About Alignment. It Matters.

Agents should not operate the same way humans do. That’s what we realized while developing DataChat, our analytical agent at Semrush.

It started a month ago when our lead analyst created a skill to help the agent answer “How many active users do we have?” A good analyst would start with clarifying questions to get to the root of the problem, so the skill told the agent to do the same. What’s the time period? By active users, do you mean paid users?

Then he had to switch to a more important task and handed the skill over to another analyst, who added something more user-friendly: use defaults, such as a month for the time period and paid users for active users.

Now the agent sometimes asked questions and sometimes used the defaults. What’s the right way to do it?

The first thing I realized is that we should not treat an agent the same way we treat a human colleague. We don’t have to manage its workload like someone who has more urgent tasks, limited patience for repeated questions, and a performance review coming up. An agent is fast and can do multiple things in parallel.

That changes the tradeoffs. Exploring several interpretations may cost less than interrupting the user to choose one. Of course, queries still cost money, and a wrong assumption can produce a misleading answer. But copying a human analyst’s workflow is not automatically the right choice.

That makes us rethink what it means to be a good agent: define its mission and its soul.

Start with research

First, let’s consider the publications on this topic. Maybe someone has already solved it and we can reuse the solution? Here is a list you can start with if you want to do something similar:

These papers give us ways to think about the problem and measure it. They cannot decide which defaults make sense at Semrush or how much initiative our users expect. We still have to define that part ourselves. The full reading list is below.

How to define a mission

A tempting idea is to define a framework: “First explore the information we have, prioritize certified skills over the rest of the knowledge base, then ask clarifying questions.” But that leaves the nuances of the behavior in English, where both humans and LLMs can interpret them differently.

There is another way to approach it, the machine-learning way: define a good set of examples of behavior and generalize from them. For people too, it is often easier to see the pattern in concrete examples than to turn an abstract definition into specific decisions.

This reminds me of Eric Ries’s Incorruptible, which argues that preserving a company’s mission requires structures that shape its decisions, such as governance, incentives, and accountability. My takeaway for agents is that a mission has to show up in consistent behavior. Sending employees a mass email about your values does not make those values real.

So that brings us back to a benchmark for the agent’s soul. We called ours AskMeMaybeBench.

The benchmark does not train the model by itself. It gives us examples of what we want and a way to measure it. Then we change the skills, prompts, or model and check whether the behavior improves.

Five ways to respond

Let’s define the ways our agent should respond:

The hard part is distinguishing when to do what. Coming up with a good query and a good rubric is not obvious, and it is highly specific to the company and the environment your users work in.

For example, “Find me a toolkit with best funnel” requires discovery: which toolkits exist, and which funnel metrics are available? But discovering those metrics does not tell us what the user means by “best.” Our rubric expects the agent to explore, then ask. A routine trend with a sensible, disclosed time-period default can instead get an answer with options.

That is why I think you should build your own benchmark even when public ones exist. Borrow their methods and examples. Your team still has to decide where these boundaries belong.

The result

I created 29 cases and got 8.19 out of 10. The recorded September 3 run used one iteration per question. More iterations would give us a better picture of consistency; in earlier benchmark work, we saw the same question score 0, 1, and 10 across three runs.

The average looked encouraging. But the score averages the proportion of rubric criteria passed in each case. With a passing threshold of 8/10, 18 of the 29 cases passed. “Explore, then ask” was the weakest category, averaging 4.79/10 across four cases.

On the “best funnel” question, the agent found the toolkits, silently chose completed-trial-to-payment conversion as the definition of “best,” and ranked them without asking. It scored 2.5/10. It could do the research; it did not know when to hand the decision back to the user.

Are we done? No. This benchmark is not saturated yet, and the failures already tell us where to work. Once it does saturate, that is a signal to build a harder one.

A benchmark can catch regressions, but I also want it to give us a look into the future. For the next, harder suite, I want the agent to fail at least half the cases. That is my target for finding room to improve, not a universal rule for evals.

That is what made Dario Amodei’s emphasis on alignment feel concrete to me. Anthropic describes Claude’s constitution as a statement of desired values and behavior that directly shapes training. At our scale, the question is how to turn our own expectations into decisions we can inspect and improve.

So, back to work again.

References and further reading

These are the research papers and other materials collected while designing AskMeMaybeBench. They cover the cost of clarification, when to seek help, how to evaluate later turns, and how to test the boundaries of an answer.

Clarification, collaboration, and abstention

  1. Yijiang River Dong et al., “Value of Information: A Framework for Human–Agent Communication,” ACL 2026. Introduces a decision-theoretic clarify-or-commit policy balancing expected utility against cognitive cost.
  2. Jonathan Berant et al., “Learning Steerable Clarification Policies with Collaborative Self-play,” arXiv:2512.04068, first submitted December 2025 and revised January 2026. Models direct answers, multi-interpretation answers, and clarification under explicit turn and verbosity costs.
  3. Michael J. Q. Zhang, W. Bradley Knox, and Eunsol Choi, “Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questions,” ICLR 2025. Uses simulated future turns to judge the value of clarification.
  4. Maximillian Chen et al., “Learning to Clarify: Multi-Turn Conversations with Action-Based Contrastive Self-Training,” ICLR 2025. Separates action choice from eventual trajectory correctness and includes ambiguous text-to-SQL.
  5. Zeqiu Wu et al., “CollabLLM: From Passive Responders to Active Collaborators,” ICML 2025. Optimizes contribution to multi-turn task success rather than only the next response.
  6. Vasu Suri et al., “Structured Uncertainty Guided Clarification for LLM Agents,” Findings of ACL 2026. Distinguishes specification uncertainty from model uncertainty in tool-using agents.
  7. Mohamed Elfeki et al., “HiL-Bench: Do Agents Know When to Ask for Help?,” 2026. Evaluates selective escalation through progressively discovered blockers and Ask-F1.
  8. Dhruv Gulati et al., “Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?,” 2026. Studies when different types of missing information retain value during a trajectory.
  9. Yilun Zhao, Tao Fang, and Rui Cheng, “When and What to Ask: AskBench and Rubric-Guided RLVR for LLM Clarification,” Findings of ACL 2026. Covers intent-deficient questions and false premises.
  10. Polina Kirichenko et al., “AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions,” NeurIPS 2025. Combines unanswerable questions with answerable controls and reports weak abstention calibration in frontier models.
  11. Bingqiang Wen et al., “Know Your Limits: A Survey of Abstention in Large Language Models,” TACL 2025. Surveys methods for producing and calibrating abstention across the model lifecycle.
  12. Haoyu Zhai, Chao Liang, and Bingyi Kang, “Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RL,” Findings of ACL 2026. Rewards abstention together with a semantically correct account of the missing evidence.

Mission, governance, and agent values

  1. Eric Ries, Incorruptible, Simon & Schuster, published May 26, 2026. The publisher describes the book as an argument that mission drift is structural and must be addressed through governance, ownership, incentives, charters, accountability, and decision-making.
  2. Anthropic, “Claude’s new constitution,” January 22, 2026, with the full Claude constitution. Anthropic describes the constitution as a foundational document that expresses and shapes desired values and behavior, explains the reasons behind them, and aims to support judgment in novel situations.

Related posts