On this page 12 sections
- Key takeaways
- Who this applies to
- Three conditions that justify splitting
- What splitting costs
- The question that settles most cases
- The shape that usually works
- How to decompose, if you are splitting
- Evaluating a multi-agent system
- What we recommend, and how often we recommend against it
- When to use neither
- Frequently asked questions
- Next step
Split into multiple agents when the sub-tasks need genuinely different tools or permissions, when they can run in parallel, or when one agent’s tool list has grown large enough to degrade selection. Otherwise a single agent, or a deterministic workflow calling a model at specific steps, is cheaper, faster and far easier to debug. Most multi-agent designs are workflows that have been given autonomy they do not need.
The question to ask of any multi-agent proposal: what is each agent deciding that could not be decided in advance?
Key takeaways
- Different permissions is the strongest reason to split. Different topics is the weakest.
- Every handoff loses context and adds latency and cost.
- Debugging cost grows faster than agent count, because failures move between them.
- If the sequence is known, it is a workflow, and workflows are testable.
- Parallelism is a real win when sub-tasks are genuinely independent.
Who this applies to
You are designing a system where a model does several things, and deciding between one agent, several coordinated agents, or a deterministic pipeline with model calls in it.
Three conditions that justify splitting
1. Different permissions or credentials
The strongest reason, and the one that is about safety rather than performance.
An agent that reads customer records and an agent that writes to the billing system should be different components with different credentials, because that bounds what each can do. Merging them creates one component holding the union of both capabilities, which is exactly what you do not want when something reasons badly.
This is the same argument as tiering tools - see tool-calling agents: what to let them touch - taken one step further into the architecture.
2. Genuine parallelism
If three sub-tasks are independent, running them concurrently converts three sequential waits into one. For anything user-facing this can be the difference between usable and not.
The condition is genuine independence. If B needs A’s output, you have a sequence, and calling it multi-agent does not make it concurrent.
3. Tool selection degrading
An agent with a very large tool list picks the wrong tool more often, because selection is itself a classification problem over a growing option space.
Splitting into agents with focused tool sets, behind a router, improves selection accuracy. The threshold varies by model, and the signal is empirical: measure tool-selection accuracy as its own metric, and if it degrades as you add tools, that is your answer.
What splitting costs
Rarely budgeted, and it is the reason most of these designs disappoint.
Context is lost at every handoff. Agent A summarises for agent B, and the summary omits something. This is the dominant failure mode in multi-agent systems and it is not fixable by better prompting - it is inherent in passing a compressed representation between components.
Latency compounds. Each agent is at least one model call. Four sequential agents is four round trips before the user sees anything.
Cost multiplies. Each agent re-reads context. A four-agent chain frequently costs several times a single agent on the same task, and the overlap is mostly re-sent context.
Debugging gets disproportionately harder. With one agent, a bad output has one place to look. With four, the failure may be in any of them or in a handoff, and the failure often moves between runs. Debugging cost grows faster than agent count.
Failure modes multiply. Agents disagreeing, looping between each other, or one confidently acting on another’s misunderstanding.
The question that settles most cases
What is each agent deciding that could not be decided in advance?
If the answer is “nothing - it always does step one, then two, then three”, that is a workflow. Write it as code that calls a model at the steps that need judgement.
A workflow is testable, debuggable, cheaper, faster and predictable. You give all of that up for autonomy, and autonomy is only worth it where the path genuinely varies by input.
Most multi-agent architectures we are asked to review are sequences someone knew in advance, implemented as agents because the framework encouraged it.
The shape that usually works
Where a split is justified, the pattern that holds up:
A deterministic orchestrator, not an orchestrating agent. Code decides which specialist runs, based on a classification step. That keeps control flow inspectable and testable. An agent deciding which agent to call adds a layer whose failures are the hardest to reason about.
Specialists with narrow tools and narrow credentials. Each does one thing with the minimum access.
Structured handoffs. Agents pass validated structured objects, not prose. Prose handoffs are where context silently goes missing - see structured output.
One place that owns the user-facing response. Multiple agents writing to the user produces inconsistent tone and duplicated information.
That design is closer to a service architecture with model calls than to a swarm, and that is the point.
How to decompose, if you are splitting
Where a split is justified, the boundary you choose determines how debuggable the result is.
Split on permission boundaries first. If two operations need different credentials, that is a natural seam and it buys you safety as well as structure.
Then on genuinely independent work that can run concurrently.
Then on tool-set size, if selection accuracy is measurably degrading.
Do not split on topic. “A billing agent and a shipping agent” is the most common decomposition and usually the worst. Topics are not independent - a customer asks about a late delivery and a refund in the same message - so you immediately need coordination between agents that could have been one agent with both tools.
The test for a proposed boundary: can you write the contract between the two components as a schema, and would a person understand the failure if only one side were wrong? If the handoff is naturally structured - a request object, a decision, a result - the boundary is real. If you find yourself passing a paragraph of context because “the next agent needs to understand the situation”, the boundary is arbitrary and you are about to lose information across it.
That test also tells you what to build first. A boundary you can express as a schema can be built and tested independently on each side, which is the only way a multi-component system stays maintainable after handover.
Evaluating a multi-agent system
Harder than a single agent, and the mistake is measuring only the end-to-end output.
Measure each agent against its own contract - given this input, did it produce the correct structured output. And measure the handoffs, because that is where the loss is. An end-to-end score of 68% tells you something is wrong; per-agent scores tell you which component.
Also track how often the orchestrator routes correctly. Misrouting is a distinct failure from a specialist performing badly, and the fixes are unrelated.
What we recommend, and how often we recommend against it
Our default is one agent, and a deterministic workflow if the path is known. We propose multiple agents when permissions genuinely differ or when parallelism is genuinely available.
The reason is what happens in month six. A single agent that misbehaves is a prompt, a tool list and a trace. A four-agent system that misbehaves is an investigation, and the client’s team is the one running it after handover. We are optimising for the system being maintainable by people who did not build it, and agent count works directly against that.
The argument we most often have: multi-agent designs demo extremely well. Watching specialists coordinate is impressive in a way that a workflow calling a model three times is not, and that impression sometimes drives the decision. The honest question is whether any agent is making a decision that could have been made in advance, and where the answer is no, the demo is the only thing the architecture bought.
Where we are genuinely wrong to resist: parallel independent sub-tasks. Research or enrichment tasks that fan out across sources are meaningfully faster split, and we have under-used that pattern in the past because of a general bias toward simplicity.
When to use neither
When one model call does it. Not everything needs an agent loop. If the task is a single transformation with a known input and output, call the model and validate the result.
When the task is deterministic. No model needed at all. A surprising share of “AI workflows” are rules with a language model in the middle for no reason.
When you cannot evaluate it. If you cannot say what correct looks like per component, do not build a system with several of them.
Frequently asked questions
Is a supervisor agent a good pattern?
Usually worse than a deterministic orchestrator. It adds a model call, non-determinism in control flow, and the hardest failure mode to debug. Use code to route and a model to classify where routing needs judgement.
How many agents is too many?
If you cannot trace a failure to one component within a few minutes, you have too many. In practice that is often three or four for a team maintaining it alongside other work.
Do agent frameworks help?
They speed up the first version and can encourage complexity the problem does not need. Useful for orchestration primitives; less useful as an architectural default. Ask what the framework is deciding for you.
How do we stop agents looping between each other?
A global step budget across the whole run, not per agent, plus a repeated-state guard. Per-agent limits do not prevent two agents passing work back and forth.
Should each agent have its own evaluation set?
Yes, against its own input and output contract, plus an end-to-end set. Without per-agent sets you cannot attribute a regression.
Next step
If you have a multi-agent design and cannot name what each agent decides that could not be decided in advance, that is worth resolving before building it. The custom AI development engagement starts with the decomposition, including the version where the answer is one agent or a workflow.
Related: Tool-calling agents: what to let them touch · Structured output: getting reliable JSON · How to measure whether an AI system works · Custom AI development