Most people still use LLMs like a single “oracle”: pick one model, ask, and move on.

Andrej Karpathy’s open-source LLM Council is a compelling alternative: treat models as a panel of experts, force them to critique one another, then have a “Chairman” synthesize the best parts into a final response.

How it works:
• Stage 1 — First opinions: every model answers independently; you can inspect each response side-by-side in a tab view.
• Stage 2 — Peer review: models see the other answers with identities anonymized, then rank them on accuracy and insight.
• Stage 3 — Chairman synthesis: one designated model compiles a single final answer from the full set of responses and rankings.

Why this matters for research:

• Faster triangulation across different reasoning styles and failure modes

• A built-in “devil’s advocate” step that surfaces missing assumptions

• A clearer audit trail: what each model said, and where the disagreements are

Practical notes: it’s a local web app (FastAPI + React) routed through OpenRouter, so you can mix providers/models, but you’ll pay per call and accept added latency. Karpathy also frames it as a “vibe coded” Saturday hack and doesn’t plan to support it long-term.

If you use LLMs for serious analysis: would you rather rely on one strong model, or a committee with peer review?

Link to the GitHub:
https://github.com/karpathy/llm-council