Agents orchestration and Subagents turned out to be a bad idea for us.

The previous post ended with a conclusion to move our analytics agent to GPT‑5.6 and lower the reasoning effort of its sub-agents. While validating that decision across all 68 questions in maratbench1 I saw an easy win.

We have an explorer sub-agent that the supervisor can call, which has all the tools main agent has. It searches BigQuery, Confluence, and OpenMetadata, then returns what it finds. Across 680 historical executions, answers that used it scored 4.35 out of 10 and took 138 seconds. Answers that did not use it scored 6.36 and took 33 seconds. The explorer fired on 38.5% of executions, so removing it appeared to offer the rarest kind of optimization: faster, cheaper, and more accurate.

The suspiciously obvious bottleneck

But the journey just began :) The supervisor calls explorer on questions that already look difficult, since evals that never trigger it average 7.03; those that always trigger it average 3.54. We were mostly measuring which questions were hard, not what the explorer did to them.

In 23% of eval-and-run combinations, the same question triggered the explorer on some repeats and not on others. That variation let us make a better comparison: match each eval against itself, under the same supervisor and in the same benchmark run.

This is not a randomized experiment, so take it with a grain of salt. But it removes the largest confound, question difficulty, and it reverses the original accuracy story: we had no evidence that the explorer was poisoning answers. The latency result was much clearer: 43 of 46 matched comparisons were slower. So, replacing it still sounds like a good idea, right?

The need survived the tool

We first disabled the explorer on ten explorer-heavy evals without changing anything else. But if the agent wants to explore then it will do that, just using the other tools he has. It rerouted from ask_explorer to ask_open_metadata and Confluence, tools that were running a much more expensive model configuration.

So going from $0.11 per explorer run we got to $1-2. Doesn't sound like a easy win, right? That is compatible with the matched estimate above, where total execution cost fell when the explorer was used: the alternatives it displaced could cost more than the call itself. The component that looked expensive in aggregate was actually the cheapest information-gathering path available.

The worst execution exposed what was happening inside the new route: one OpenMetadata call consumed 7.75 million tokens and cost $7.23; the execution still scored zero. Our supervisor trace showed fewer calls than the baseline, so nothing at that level explained the bill.

The route includes the model

It was a bug - OpenMetadata was running GPT‑5.6 Terra at high effort. Moving it to Luna at low effort reduced that segment from 1.2 million to 397 thousand tokens per execution and from $1.08 to $0.238. Now we are getting somewhere.

“Which agent?” and “which model?” were not separate architecture questions after all. Uno-Orchestra frames routing in the same order our incident forced on us: first decide whether to delegate, then jointly choose the worker and model under a cost constraint. We had removed one worker while leaving its substitutes on a radically different model configuration. That changed the economics more than removing the component itself.

With cost neutralized, we could compare the information paths on more equal terms. In these rerouting runs, OpenMetadata and Confluence added about 31 seconds over an execution without sub-agents. The explorer added 119 seconds, compared with 93 seconds in the broader matched analysis, while making fewer observable tool calls. Its long internal loop was a detour, not a unique capability.

Disaster detector

The ten-question arm compared one explorer-off execution per eval with six prior baseline executions. It gave us a strong cost alarm and an almost meaningless accuracy result. Across the benchmark, the within-eval standard deviation is 2.87 points. Half of the evals are effectively coin flips: the same question can score 0, 1, 8, 10, 10, 10 across repeats. Twice that standard deviation, roughly 5.7 points, gives a sense of how large the noise is at the level of one execution; it is not a universal significance threshold.

In the ten-question arm, nine results landed inside their eval's historical range. The apparent breakthroughs and regressions were therefore inconclusive: one new trial compared with the average of six noisy ones. The small arm was good at catching a tripled bill. It could not prove a one-point accuracy gain.

This is a general property of agent evals, not a defect unique to ours. As Anthropic's guide to agent evals emphasizes, each attempt is a stochastic trial. Repeats can tell you whether an agent ever succeeds or whether it succeeds reliably, but they do not make a small comparison powerful enough by themselves. Three repeats are enough to maintain our benchmark; they are not always enough to settle a close architecture decision on a narrow slice.

That is why getting to the final conclusion took seven experimental runs. The small ones found failure modes. To measure the architecture, we had to return to all 68 questions, three repeats each.

Fixed it

The explorer had broad tools, a fresh context, and permission to keep looking. Its real architectural benefit was context isolation: it could do noisy research without filling the supervisor's context, then return a summary. What it did not have was unique expertise. Once the same evidence was reachable through direct tools and bounded specialists, the long delegated loop had little left to justify its cost.

Research on sub-agents makes the same boundary visible from both sides. SearchSwarm found that simply exposing a sub-agent tool helped broad information-seeking only a little; a complete harness with explicit decomposition, worker briefs, citations, and supervisor verification helped much more. Meanwhile, an equal-token comparison on multi-hop reasoning found that single agents matched or beat multi-agent variants once both received the same thinking budget. Neither study settles enterprise analytics. Together they suggest a useful burden of proof: delegation should earn its extra boundary through unique capability, useful isolation, or parallel work, and it should beat direct access at a comparable budget.

Our explorer had the wrong shape for that advantage. It performed one long serial search while the supervisor waited. So the final configuration changed the whole route: disable the explorer, expose the dbt repository directly to the supervisor so it could inspect transformation logic without delegation, and move the remaining sub-agents to cheaper models.

Accuracy rose by 0.51 points, with a 95% confidence interval from +0.12 to +0.93. Latency fell 22%, cost fell 68%, and eight more questions passed. Most importantly, the 22 evals that had historically used the explorer at least 70% of the time improved by 0.64 points and became 32 seconds faster. If we had removed a unique capability, this was where the damage should have appeared. It did not.

The attribution caveat matters. Explorer removal, direct dbt access, and model reassignment changed together between the baseline and final full runs. The package is clearly good enough to ship, but the +0.51 accuracy gain cannot honestly be credited to any single part. The dbt tool appeared in only 12 of 204 executions, so even its contribution remains weakly evidenced.

Wrong mechanism, right decision

The original hypothesis was right about the decision and wrong about the mechanism. The explorer was not poisoning answers; it was a slow path through information the supervisor could obtain more directly. The component that tripled the bill was not the explorer either, but the model behind the route that replaced it. The eventual win came from redesigning the substitutes, not merely deleting one box.

And the ceiling from the previous post is still there. Hard-question accuracy moved from 4.29 to 4.10. The same dead domains still need correct metric definitions, table documentation, and skills. We optimized the delivery path, not the knowledge.

Causal Agent Replay formalizes the distinction we stumbled into: observability tells you what happened, while causal attribution requires changing a step and replaying the stochastic downstream trajectory enough times to estimate an outcome distribution. That is a stricter standard than our coarse architectural intervention, and the right direction for the next experiment.

That is the practical lesson I am taking from the experiment: in an agent system, components are substitutes, not isolated boxes. Removing one route changes traffic everywhere else. Correlation can identify where to investigate, but only intervention reveals where the work goes next. And if sub-agent internals are missing from your traces, the thing you most need to optimize may be the thing your dashboard cannot show you.

References

• Boqin Yuan et al., “ClawTrace: Cost-Aware Tracing for LLM Agent Skill Distillation” (2026).
• Zhiqing Cui et al., “Uno-Orchestra: Parsimonious Agent Routing via Selective Delegation” (2026).
• Anthropic, “Demystifying evals for AI agents” (2026).
• Dat Tran and Douwe Kiela, “Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets” (2026).
• Pu Ning et al., “SearchSwarm: Towards Delegation Intelligence in Agentic LLMs for Long-Horizon Deep Research” (2026).
• Jaineet Shah, “Causal Agent Replay: Counterfactual Attribution for LLM-Agent Failures” (2026).