What does the Society of Minds protocol have N model instances do?

Independently propose answers, then over R rounds read and critique each other's proposals until they converge

Each runs a different benchmark

Why does cross-model debate (e.g. ChatGPT + Bard) outperform single-model debate?

Heterogeneity reduces shared blind spots so cross-critique catches more errors

Different vendors negotiate prices

What is the main token-cost win of a sparse topology over full mesh?

Each debater reads only a subset of peers, so critique-op count drops while accuracy often matches

Sparse topology doubles the rounds

Sparse topology requires no models

Which case does the lesson list as where debate hurts?

A simple factual lookup, because one lookup is cheaper than five debates

What is convergence collapse?

All agents converge on the first wrong answer; mitigate with required disagreement rounds

Why does prompt homogenization undermine debate?

Identical prompts produce nearly identical answers, removing the cross-critique signal

Which production pattern is a debate variant per the lesson?

Anthropic orchestrator-workers with a synthesis step

Multi-Agent Debate and Collaboration

Du et al. (ICML 2024, "Society of Minds") run N model instances that independently propose answers, then iteratively critique each other over R rounds to converge. Improves factuality, rule-following, reasoning. Sparse topology beats full mesh on token cost.

Lesson 25 of 54Learn + Build~60 minutesPython (stdlib)

Before this: Lesson 12 of Phase 14 (Workflow Patterns), lesson 5 of Phase 14 (Self-Refine and CRITIC)

On this page

Learning Objectives

Explain the debate protocol: N proposers, R rounds, converge on a shared answer.
Describe why debate improves factuality, rule-following, and reasoning.
Explain sparse topology: not every debater needs to see every other.
Implement a stdlib debate over a scripted LLM with full-mesh and sparse variants; measure token cost vs accuracy.

The Problem

Self-Refine (Lesson 05) is one model critiquing itself — risks groupthink. CRITIC (Lesson 05) grounds critique in external tools — not always available. Debate introduces a third mode: multiple instances, cross-critique, convergence by disagreement.

The Concept

Society of Minds (Du et al., ICML 2024)

N model instances independently propose answers to the same question.
Over R rounds, each model reads the others' proposals and critiques them.
Models update their answers based on the critiques.
After R rounds, return the convergent answer.

Original experiments used N=3, R=2 due to cost. Accuracy improves with more agents and more rounds on hard problems (MMLU, GSM8K, Chess Move Validity, biography generation).

Cross-model combinations beat single-model debates: ChatGPT + Bard together > either alone.

Sparse topology

"Improving Multi-Agent Debate with Sparse Communication Topology" (arXiv:2406.11776, 2024-2025) showed full-mesh debate is not always optimal. Sparse topologies (star, ring, hub-and-spoke) can match accuracy at lower token cost. Each debater sees only a subset of peers.

Implications:

Full mesh N=5, R=3 = 5 × 3 = 15 proposals, each reading 4 peers = 60 critique ops.
Star N=5, R=3 (one hub + 4 spokes) = 15 proposals, spokes read only the hub = 12 critique ops.

When debate helps

Factuality. N independent proposals, cross-check reduces hallucination.
Rule-following. Chess move validity — one model misses a rule, others catch it.
Open-ended reasoning. Multiple framings narrow in on the right answer.

When debate hurts

Latency-sensitive UX. N × R serial rounds is latency you may not have.
Cost-sensitive scale. N × R tokens per question.
Simple factual lookups. One lookup is cheaper than five debates.

2026 practical instantiations

Anthropic orchestrator-workers (Lesson 12) — one variant of debate with a synthesis step.
LangGraph supervisor (Lesson 13) — central router + specialist agents can implement debate as a node.
OpenAI Agents SDK (Lesson 16) — agents handoff back and forth for iterative critique.
Multi-agent evals — pair debate + evaluator-optimizer for eval signal.

Where this pattern goes wrong

Convergence collapse. All agents converge on the first wrong answer. Mitigate with required disagreement rounds.
Hub failure. In a star topology, a bad hub corrupts everyone. Rotate or use multiple hubs.
Prompt homogenization. All agents use the same prompt; they produce the same answers. Use diverse prompts and/or models.

debate-converge

Build It

code/main.py implements stdlib debate:

Debater class (scripted LLM with per-debater opinion drift).
FullMeshDebate and SparseDebate runners.
Three questions: one factual, one rule-based, one reasoning.
Metrics: convergent answer, rounds to convergence, total critique ops.

Run it:

python3 code/main.py

Output: per-protocol accuracy and cost; sparse matches full mesh on 2/3 questions at lower cost.

Use It

Anthropic orchestrator-workers for simple 2-3-worker debates.
LangGraph for stateful multi-round debate with checkpointing.
Custom for research or specialized correctness guarantees.

Ship It

outputs/skill-debate.md scaffolds a multi-agent debate with configurable topology, N, R, and a convergence rule.

Exercises

Implement a "forced disagreement" rule: in round 1, every debater must produce a distinct proposal. Measure effect on convergence speed.
Add a confidence-weighted aggregation: debaters return (answer, confidence); aggregator weights by confidence. Does it help?
Swap one "agent" for a different scripted LLM with different opinions. Does heterogeneity improve accuracy?
Measure token cost for full mesh vs sparse on your 3 questions. Plot cost vs accuracy.
Read the Society of Minds paper. Port your toy to N=5, R=3. What breaks? What gets better?

Key Terms

Term	What people say	What it actually means
Debate	"Multi-agent critique"	N proposers, R rounds of cross-critique, converge
Full mesh	"Everyone reads everyone"	Every debater reads every peer each round
Sparse topology	"Limited peer view"	Debaters read only a subset of peers
Hub-and-spoke	"Star topology"	One central debater, N-1 spokes read only the hub
Convergence	"Agreement"	Debaters converge on a shared answer
Society of Minds	"Du et al. debate paper"	ICML 2024 multi-agent debate method

Quick quiz

What does the Society of Minds protocol have N model instances do?
1. Independently propose answers, then over R rounds read and critique each other's proposals until they converge
2. Train on a shared dataset
3. Each runs a different benchmark
4. Negotiate prices
Show the answer
A. Independently propose answers, then over R rounds read and critique each other's proposals until they converge N proposers, R rounds, cross-critique, convergence is the canonical Du et al. debate.
Why does cross-model debate (e.g. ChatGPT + Bard) outperform single-model debate?
1. It is cheaper
2. Different vendors negotiate prices
3. Vendors share weights
4. Heterogeneity reduces shared blind spots so cross-critique catches more errors
Show the answer
D. Heterogeneity reduces shared blind spots so cross-critique catches more errors Mixing models brings independent error distributions, which raises the ceiling of debate.
What is the main token-cost win of a sparse topology over full mesh?
1. Each debater reads only a subset of peers, so critique-op count drops while accuracy often matches
2. It removes the LLM
3. Sparse topology doubles the rounds
4. Sparse topology requires no models
Show the answer
A. Each debater reads only a subset of peers, so critique-op count drops while accuracy often matches Sparse (star, ring, hub-and-spoke) reduces critique ops without losing accuracy on many tasks.
Which case does the lesson list as where debate hurts?
1. A simple factual lookup, because one lookup is cheaper than five debates
2. Open-ended reasoning
3. Chess move validity
4. Biography generation
Show the answer
A. A simple factual lookup, because one lookup is cheaper than five debates Latency- and cost-sensitive trivial lookups do not benefit from N x R debate.
What is convergence collapse?
1. Process crash
2. TLS handshake failure
3. Network outage
4. All agents converge on the first wrong answer; mitigate with required disagreement rounds
Show the answer
D. All agents converge on the first wrong answer; mitigate with required disagreement rounds Early agreement on a wrong answer is mitigated by forcing distinct round-1 proposals.
Why does prompt homogenization undermine debate?
1. It violates Apache 2.0
2. It bypasses the supervisor
3. Identical prompts produce nearly identical answers, removing the cross-critique signal
4. It uses too many tokens
Show the answer
C. Identical prompts produce nearly identical answers, removing the cross-critique signal Diverse prompts (and ideally diverse models) keep the proposal distribution wide.
Which production pattern is a debate variant per the lesson?
1. Pure single-shot RAG
2. Vector indexing
3. Anthropic orchestrator-workers with a synthesis step
4. Cron-only schedule
Show the answer
C. Anthropic orchestrator-workers with a synthesis step Orchestrator-workers with synthesis is a debate-shaped pattern in production.

Questions people ask

Does having multiple AI models debate improve accuracy?

Research shows debate can improve factual accuracy and reasoning on many tasks, because models catch each other's mistakes. It is not perfect: models can also talk each other into a shared wrong answer, and cost grows with each agent and round.

How many AI agents should debate a question?

A few agents over a couple of rounds is a common start. More agents and rounds raise cost quickly. Letting each agent see only some peers, instead of everyone, can keep most of the benefit at lower cost.

Run the code Read it on GitHub

Next lessonFailure Modes: Why Agents Break Previous: Agent Observability: Langfuse, Phoenix, Opik

Lessons from AI Engineering from Scratch by Rohit Ghumare and contributors, MIT license. Notes for owners by Lozo. License