Building a Multi-Agent System: Patterns and Pitfalls

Once a single agent works, the next idea is usually to build a team of them: a planner, a researcher, a writer, a reviewer, each with its own prompt and tools. It looks like an org chart, and it is easy to sketch.

It is also easy to get wrong. A multi-agent system adds hand-offs, and every hand-off is a place where context is lost and errors pass along. Some teams report large gains. Others, with equal experience, advise against the whole idea. Both are right about different tasks.

What the evidence says

Anthropic built its research feature as a lead agent that plans and then starts subagents to search in parallel. On its internal research evaluation, that system outperformed a single agent using the stronger model by 90.2%. Its analysis found the main reason was simple: the extra agents let the system spend more tokens on the problem. Token usage alone explained 80% of the difference in performance on a browsing benchmark (Anthropic: multi-agent research system).

The same write-up is candid about the price. Agents used about four times the tokens of a chat, and multi-agent systems about fifteen times. Tasks where all agents need the same context, or with many dependencies between agents, fit poorly. Most coding tasks are an example.

Cognition, which builds a coding agent, makes the opposite case. Subagents working from partial context make decisions that conflict. A final agent is then left to combine pieces that were built on different assumptions. Its principles are to share full context and traces between steps, and to remember that every action carries implicit decisions. The simplest architecture that obeys both is a single agent working through the task in order (Cognition: Don't Build Multi-Agents).

Research agrees that the gains are not automatic. A study of more than 1,600 annotated traces from seven multi-agent frameworks found performance gains on popular benchmarks often minimal. It sorted the failures into 14 modes in three groups: system design issues, misalignment between agents, and weak verification of the result (Cemri et al.: Why Do Multi-Agent LLM Systems Fail?).

The shared conclusion: multiple agents help when the work splits into parts that are genuinely independent, such as searching many sources at once, and when it is valuable enough to pay for the tokens. They hurt when the parts depend on each other.

The patterns

Several of these do not need more than one agent at all. A pipeline or a critic loop can be ordinary code calling a model several times, which is easier to debug.

How errors compound

The script is a small probability model, not a measurement. It asks three questions. How reliable is a chain of hand-offs? How much does a checker after each step recover? And how much does voting help when agents fail in the same way?

# How reliability behaves when work passes between agents.
# A simple probability model: each step succeeds or fails independently unless stated otherwise.

def chain(p, steps):
    # every hand-off must succeed for the final result to be right
    return p ** steps

print("1. A chain of agents, each step right with probability p")
print(f"{'steps':>5} {'p = 0.99':>9} {'p = 0.95':>9} {'p = 0.90':>9}")
for steps in (1, 3, 5, 10, 20):
    row = " ".join(f"{chain(p, steps):>9.1%}" for p in (0.99, 0.95, 0.90))
    print(f"{steps:>5} {row}")

# A checker after each step catches a wrong result with probability `catch`,
# and the step is then redone once, with the same chance of success.
def checked_step(p, catch):
    return p + (1 - p) * catch * p

print()
print("2. Ten steps at p = 0.95, with a checker that triggers one retry")
print(f"{'checker catches':>16} {'per step':>9} {'all ten':>8}")
for catch in (0.0, 0.5, 0.8, 0.95):
    step = checked_step(0.95, catch)
    print(f"{catch:>16.0%} {step:>9.2%} {chain(step, 10):>8.1%}")

# Three agents answer the same question and the majority wins.
# With probability `shared`, a common cause (same model, same prompt, same
# misleading source) makes all three right or all three wrong together.
def majority_of_three(p, shared):
    independent = p ** 3 + 3 * p ** 2 * (1 - p)
    return shared * p + (1 - shared) * independent

print()
print("3. Majority vote of three agents, each right 80% of the time")
print(f"{'shared errors':>14} {'vote right':>11}")
for shared in (0.0, 0.25, 0.5, 0.75, 1.0):
    print(f"{shared:>14.0%} {majority_of_three(0.8, shared):>11.1%}")

1. A chain of agents, each step right with probability p
steps  p = 0.99  p = 0.95  p = 0.90
    1     99.0%     95.0%     90.0%
    3     97.0%     85.7%     72.9%
    5     95.1%     77.4%     59.0%
   10     90.4%     59.9%     34.9%
   20     81.8%     35.8%     12.2%

2. Ten steps at p = 0.95, with a checker that triggers one retry
 checker catches  per step  all ten
              0%    95.00%    59.9%
             50%    97.38%    76.6%
             80%    98.80%    88.6%
             95%    99.51%    95.2%

3. Majority vote of three agents, each right 80% of the time
 shared errors  vote right
            0%       89.6%
           25%       87.2%
           50%       84.8%
           75%       82.4%
          100%       80.0%

• Chains decay fast. A step that is right 95% of the time sounds dependable. Ten of them in a row deliver a correct result 59.9% of the time, and twenty only 35.8%.
• Small gains per step matter. At 99% per step, ten steps still give 90.4%. Improving each agent is worth more than adding agents.
• Checking is the strongest lever. A checker that catches 80% of bad results and triggers one retry lifts the ten-step chain from 59.9% to 88.6%. This is the "task verification" category of failures from the research above, addressed directly.
• Voting helps only with independent errors. Three agents that are each right 80% of the time vote correctly 89.6% of the time if their mistakes are unrelated. If they share the same model, prompt and sources, their mistakes overlap, and when they fail together every time the vote is no better than one agent: 80.0%.

Real agents are not this tidy. Steps are not independent, and some errors are harmless. But the direction holds: more hand-offs mean more ways to fail, unless something checks the work.

Design rules

1. Start with one agent. Move to several only when you can name the independent parts and show on a test set that splitting helps. See AI agents explained.

2. Delegate like a manager, not a slogan. Anthropic found each subagent needed an objective, an output format, guidance on tools and sources, and clear boundaries. Without them, agents duplicated work and left gaps.

3. Scale effort to the task. Early versions of Anthropic's system started 50 subagents for simple queries. It fixed this with explicit rules in the prompt: one agent with 3 to 10 tool calls for a simple fact, more than 10 subagents only for complex research.

4. Share context, not summaries. Pass the full task and the relevant history to each agent, or have agents write results to a shared store that the next one reads directly.

5. Verify at every hand-off. A schema check, a test, or a critic agent with clear criteria. See evaluating LLM outputs.

6. Budget and stop. Set limits on agents, steps, tool calls and tokens, and make the system stop and report when it reaches one. See cost optimization for LLM applications.

7. Log everything. Every prompt, tool call and hand-off, so that a failure can be traced to the agent and step that caused it.

Frameworks

Frameworks give you the plumbing: message passing, hand-offs, state and retries. AutoGen, from Microsoft Research, builds applications from agents that converse with each other and with people and tools (Wu et al.: AutoGen). OpenAI's Agents SDK has a runner that handles the agent loop and hand-offs between agents inside your application (OpenAI: agents). Others, such as CrewAI and LangGraph, take their own approaches.

None of them decides how to split the work, and that is the decision that determines success. See the agentic AI roadmap for where frameworks fit in learning.

Common mistakes

• Designing the org chart first and the task decomposition second.
• Agents that talk in summaries, losing the details the next one needed.
• No verification between steps, so a wrong early result reaches the end.
• Voting among copies of the same model and prompt and expecting independent judgements.
• No limits, so a confused system spawns agents and spends tokens without end.
• Splitting a coding task into parts that each make different assumptions about the whole.

Multi-agent frameworks such as CrewAI are covered in the Gen AI Frameworks module of the Generative AI course.

Sources

• How we built our multi-agent research system (Anthropic) — Anthropic
• Why Do Multi-Agent LLM Systems Fail? (arXiv) — Cemri et al.
• Don't Build Multi-Agents (Cognition) — Cognition
• AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation (arXiv) — Wu et al.
• Agents (OpenAI API documentation) — OpenAI