759F233A

BUT there's a level most people are missing.

• 1.3 million views on Ole Lehmann's article alone.
• thousands of people running it to make real decisions.
• the idea is genuinely brilliant:

instead of asking one AI a question and trusting whatever comes back, you force multiple perspectives to argue, review each other's blind spots, and produce a verdict that's harder to disagree with.

Ole's implementation works. he made a real business decision with it. I'm not here to trash it. but I found a way to take it three levels further. multi-model diversity + customizable analysis lenses + a Stanford research technique that increases output diversity by 2-3x. all in one Skill. here's how it works and why each layer matters.

where this started

Andrej Karpathy (co-founder of OpenAI, former head of AI at Tesla) built the original LLM Council in November 2025. his setup:

four different models. different training data. different architectures. different blind spots. Karpathy noticed something in testing: the models consistently praised GPT-5.1 as the most insightful, but he personally preferred Gemini's condensed output. the models disagreed with the human. the human disagreed with the models. that tension is where the value lives. 17K stars on GitHub. VentureBeat called it "a blueprint for enterprise AI orchestration."

• send the same question to GPT, Claude, Gemini, and Grok simultaneously
• each model responds independently
• each model reviews the others' responses anonymously
• a chairman model synthesizes the final answer

> what happened next

Ole Lehmann rebuilt the council to run entirely inside Claude. instead of four different models, five "thinking styles" as sub-agents:

each advisor responds → anonymous peer review → chairman synthesizes. it works. for quick brainstorming, exploring angles, stress-testing ideas — it's a real improvement over asking one model one question. but two things changed in the translation: 1. model diversity disappeared. four different models → one model playing five roles. 2. customization appeared. Ole added something Karpathy didn't have: specific analytical lenses that shape HOW the model thinks about your problem. so Ole gained something valuable (customizable lenses) and lost something valuable (real model diversity). the question I kept asking: what if you could have both? and then I found a third layer.

• the Contrarian
• the First Principles Thinker
• the Expansionist
• the Outsider
• the Executor

> layer 1: why multi-model matters (the research)

there's a well-documented phenomenon called self-preference bias. three papers from top venues: Paper 1 — NeurIPS 2024"LLM Evaluators Recognize and Favor Their Own Generations" 🔗 https://arxiv.org/abs/2404.13076 LLMs score their own outputs higher than outputs from other models — even when humans consider them equal quality. linear correlation: the better a model recognizes its own output → the stronger its self-preference bias. Paper 2 — ICLR 2025"Self-Preference Bias in LLM-as-a-Judge" 🔗 https://arxiv.org/abs/2410.21819 the mechanism: models favor outputs stylistically familiar to them (measured by perplexity). closer to their own style = higher scores. Paper 3 — arXiv, April 2026"Self-Preference Bias in Rubric-Based Evaluation" 🔗 https://arxiv.org/abs/2604.06996 the bias persists even with entirely objective criteria. judges were up to 50% more likely to incorrectly mark outputs as correct when the output was their own. important nuance: this research is about the EVALUATION step. it doesn't prove single-model persona generation is useless — asking Claude to think as a Contrarian vs an Expansionist DOES produce different outputs. the gap is in the peer review step. Claude reviewing Claude's five outputs can't meaningfully differentiate because they all "feel" equally familiar. the academic recommendation: use multiple different models to lessen self-enhancement bias. Perplexity Computer's Model Council does exactly this:

that's layer 1: real model diversity instead of simulated diversity.

• routes to three frontier models simultaneously (GPT-5.4, Claude Opus 4.6, Gemini 3.1 Pro)
• each responds independently
• a fourth synthesizer analyzes convergences, divergences, unique contributions

> layer 2: why customization matters

Karpathy's council is powerful but fixed. you can't change the analytical framework. Ole's council lets you define HOW the model thinks. that's genuinely valuable. I built a custom Skill for Computer that lets you define your OWN lenses — and they run on top of Model Council's multi-model output. if you're an investor, your lenses might be:

if you're a content creator:

if you're a founder:

the Skill is just a markdown file. you write the lenses that match YOUR decision type. Model Council provides the multi-model diversity underneath. Karpathy built the engine. Ole built the dashboard. this lets you redesign the dashboard for your exact use case while keeping the engine. that's layer 2.

• Bull Case / Bear Case / Macro Factors / Portfolio Fit
• Audience Fit / Distribution Strategy / Monetization Path / Longevity Test
• Customer Demand / Technical Feasibility / Market Timing / Competitive Edge

> layer 3: the technique that changes everything

this is the part that genuinely surprised me. there's a Stanford research paper by Jiayi Zhang, Christopher Manning, and colleagues called Verbalized Sampling. 🔗 https://arxiv.org/abs/2510.01171 the problem they solve: after alignment training (RLHF, DPO), LLMs suffer from mode collapse. they converge on safe, typical, "most likely" responses. the creative, unusual, high-value insights get suppressed because the model learned that familiar = preferred. this is a deep problem. it means when you ask any AI for analysis, you're getting the most TYPICAL analysis, not the most INSIGHTFUL one. the tails of the distribution, where the genuinely surprising insights live, get cut off. their solution is deceptively simple: instead of asking the model for one answer, you ask it to generate multiple responses AND assign probability scores to each. then you explicitly tell it to sample from the tails — responses with probability less than 0.10. the result: 1.6-2.1x diversity improvement in creative tasks while maintaining quality. training-free. works on any model. orthogonal to temperature (meaning it stacks on top of temperature settings, it doesn't replace them). why this matters for council-style analysis: without VS, each model gives you its most typical response. with VS, each model explores its own distribution more broadly, surfacing insights it would normally suppress. when you combine this with multi-model (each model has a DIFFERENT distribution to explore) and custom lenses (each perspective is structured for YOUR domain), you get three layers of diversity working simultaneously: 1. between-model diversity → different training data, different knowledge 2. within-model diversity → verbalized sampling unlocks each model's suppressed insights 3. analytical diversity → custom lenses focus the output on YOUR decision type nobody has combined all three. this is the stack I'm running.

> the Stress Test Skill (with all three layers)

setup: 1. Computer → Skills → + Create Skill 2. paste the Skill below, name it "Stress Test" 3. activate Model Council (click "+" next to search bar → "Model Council") 4. type "stress test this:" + your decision with full context to customize: replace Phase 2 lenses with whatever fits your decisions.

• investor? → Bull Case / Bear Case / Macro / Portfolio Fit
• founder? → Customer / Technical / Timing / Competition
• creator? → Audience / Distribution / Monetization / Longevity

> example output

stress testing: should I gate my newsletter behind a paid tier or keep free with sponsorships? THE QUESTION: should I gate my newsletter behind a paid tier or keep it free with sponsorships? WHERE PERSPECTIVES AGREE: both paths viable at 99K subscribers. the deciding factor isn't revenue model — it's whether I optimize for growth or revenue per subscriber. three models converged independently. WHERE PERSPECTIVES DISAGREE: GPT calculated sponsorship at my scale ($3-6 CPM on 99K) = $297-594/send. paid tier at $10/month with 3% conversion = $29,700/month. 50x difference. Claude pushed back: converting free to paid requires fundamentally different content that could break what makes the newsletter work. Gemini proposed a hybrid nobody asked about: core content free, monthly deep-dive paid. tail-distribution insight (from verbalized sampling): one low-probability response flagged that at 99K subscribers, the newsletter itself might not be the revenue product - it's the trust asset. the monetization layer should sit ABOVE the newsletter (courses, consulting, community) not inside it (paywalls). the newsletter's job is attention and trust, not direct revenue. RISK: launching paid without testing conversion kills free growth AND produces insufficient paid revenue. BLIND SPOT: framing as "free vs paid" when the real question might be "is the newsletter the product or the distribution channel?" OPPORTUNITY: hybrid tests paid demand with zero risk to free base. but the tail insight suggests a third path: newsletter stays free forever, monetization happens through products the newsletter sells. VERDICT: don't paywall the newsletter. run a 4-week experiment selling a $49 product to the list instead. if it converts, the newsletter is a distribution channel and should stay free. if it doesn't, then test paid tier as fallback. TEST IT THIS WEEK: one product offer to the full list. measure revenue per subscriber. compare against sponsorship CPM. let the data decide the newsletter's role. that tail-distribution insight — "the newsletter is the trust asset, not the revenue product" — is exactly the kind of analysis that gets suppressed by default mode collapse. normal prompting would have given me the obvious "free vs paid" analysis. verbalized sampling surfaced the reframe.

> the honest summary

single-model councils (Ole's approach):

multi-model + custom Skill + verbalized sampling (what I'm running):

when to use which: → quick brainstorming, low-stakes: Ole's single-model council. fast, free, useful. → high-stakes decisions: Model Council + custom Skill + verbalized sampling on Computer. three layers of diversity. the research-backed stack. the innovation isn't just "use multiple models." it's three layers: 1. between-model diversity → Model Council routes to different models with different knowledge 2. within-model diversity → verbalized sampling unlocks each model's tail distribution 3. analytical diversity → your custom Skill focuses the output on YOUR decision type Karpathy built the engine. Ole built the dashboard. this combines both, adds a research-backed diversity amplifier, and lets you redesign everything for your exact use case.

• ✅ useful diverse perspectives through persona prompting
• ✅ fast, any Claude subscription
• ❌ peer review suffers from self-preference bias (NeurIPS 2024, ICLR 2025, arXiv 2026)
• ❌ fixed advisor types
• ❌ no verbalized sampling (mode-collapsed outputs)
• ✅ genuine model diversity (different training data, architectures, alignment)
• ✅ reduced evaluation bias (documented)
• ✅ fully customizable lenses for your domain
• ✅ verbalized sampling: 2-3x diversity, surfaces suppressed insights
• ✅ three layers of diversity working simultaneously
• ❌ verbosity and ordering bias still exist
• ❌ requires Perplexity Computer ($200/month Max or $20/month Pro)

> the research:

follow me @alex_prompter for more. I test the tools so you don't have to. P. S. check out my free Perplexity mastery guide if you want. | https://x.com/alex_prompter/status/2043011383457964460 |

• 🔗 https://arxiv.org/abs/2404.13076 (NeurIPS 2024 — self-preference bias)
• 🔗 https://arxiv.org/abs/2410.21819 (ICLR 2025 — familiarity mechanism)
• 🔗 https://arxiv.org/abs/2604.06996 (arXiv 2026 — objective criteria bias)
• 🔗 https://arxiv.org/abs/2510.01171 (Stanford — verbalized sampling, 2-3x diversity)