Superforecasters, AI Models, and Markets: Who Calibrates Best?
Philip Tetlock ran a 20-year experiment. He asked thousands of experts to make probabilistic predictions about future events. Political scientists. Intelligence analysts. Economists with decorated track records. Then he tracked whether they were right.
Most of them were not. The average expert was barely better than random. They were confident. They gave detailed reasons. And they were wrong at roughly the same rate as someone who had simply guessed.
The finding that changed forecasting research was not the failure rate. It was the exception: roughly 2% of participants were genuinely, measurably, reproducibly good at this. Not because they knew more than the experts. Because they thought differently.
Those are the superforecasters. And in 2026, with prediction markets processing billions of dollars and AI models generating probability estimates for any question you can phrase, the calibration comparison between those three methods is no longer academic. It tells a trader where to look for independent probability reference points, and when to trust each one.
Quick Answer
On long-horizon structured questions, liquid prediction markets and superforecasters calibrate at approximately the same level, Brier scores of 0.14-0.18 for both versus 0.19-0.24 for large language models. On short-horizon events where information timing matters, prediction markets outperform both. The trader’s practical stack: prediction market prices as the primary reference, superforecaster methodology as personal analytical discipline, and AI models as research tools on long-horizon thin-market questions where the crowd has not yet assembled.
Key Takeaways
- Philip Tetlock’s Good Judgment Project established that superforecasters sustain Brier scores of 0.14-0.16 on structured long-horizon questions. That is the human forecasting benchmark. It took a decade of controlled research to establish, and it holds up.
- Liquid prediction markets on comparable questions score 0.15-0.18 on Brier score. They match superforecasters on accuracy while updating in minutes rather than hours. Speed is the structural advantage, not precision.
- Large language models on equivalent question sets score 0.19-0.24 in research studies, below both superforecasters and liquid prediction markets. The gap is largest where prediction markets are strongest: short-horizon binary events where information from the past 24 hours is decisive. LLMs have knowledge cutoffs. Markets do not.
- The comparison that matters for a trader is not which method is smartest but under which conditions does each method hold information the market price does not. A thin Polymarket market on a niche question may be less calibrated than a superforecaster’s estimate of the same question. A major political market with $50 million in volume is almost certainly better calibrated than any model.
- Superforecaster methodology is directly applicable to personal trading. Decompose questions into components. Estimate base rates before current context. Track your own calibration by category over 50 or more positions. Update quickly when new evidence arrives. These habits are what separates the 2% from everyone else, in forecasting tournaments and in prediction market trading.
- The ranking is not fixed. LLM accuracy has been improving fast. Build the analytical framework around structural advantages, not around current Brier scores that will shift.
What Made Superforecasters Different
The expert failure rate in Tetlock’s study was not evenly distributed. The people who failed badly were mostly the confident ones: pundits with platforms, specialists with narrow credentials, analysts who had built career identities around being right about a domain.
What the superforecasters did differently was almost entirely methodological.
They converted vague questions into precise ones before answering. “Will there be economic disruption in Asia next year?” became “What is the probability that GDP in at least one G20 Asian nation contracts by more than 2% in the next 12 months?” Precision forces an honest probability estimate. Vagueness lets you feel correct regardless of the outcome.
They started with base rates before adding context. Before incorporating anything about a specific country’s current economic situation, they asked: how often has this type of event historically occurred in comparable circumstances? The base rate is the anchor. Current context adjusts it. Most forecasters do it backwards.
They tracked their predictions and updated their models when they were wrong. Not defensively. Analytically. Wrong calibration in one question type meant adjusting the model, not rationalising the result.
The Brier score formalises this. A perfect forecaster scores 0.00. A forecaster who always says 50% scores 0.25. The superforecaster average of 0.14-0.16 represents genuine skill sustained over hundreds of forecasts. This is the benchmark.
The 3-Way Calibration Comparison
Where superforecasters hold the edge
Thin-market specialised questions. When a Polymarket market on a niche policy question has $15,000 in total volume and the participants are not domain experts, a well-calibrated superforecaster with relevant knowledge may produce a more accurate probability estimate than the market price. The market aggregates whoever showed up. The superforecaster applies systematic methodology.
Speed is the trade-off. A superforecaster who reads an FOMC statement at 2:05 PM has a calibrated view by 2:30 PM. The prediction market has already repriced by 2:08 PM. For anything time-sensitive, human speed is the binding constraint.
Where prediction markets hold the edge
Real-time information incorporation. A Polymarket match winner market two hours before kick-off has already priced the injury report released 45 minutes ago, the pre-match press conference that ended 20 minutes ago, and any suspicious line movements from sharp sportsbooks in the last hour. No superforecaster can access and process all of that simultaneously in real time.
Scale and diversity. Major prediction markets aggregate thousands of independent views. The information aggregation theorem works better with more participants. Superforecasters number in the hundreds globally. Polymarket participants on major events number in the thousands. That scale difference matters when the information is distributed across many participants. The Market Efficiency guide covers when this aggregation breaks down and prices fail to correct.
Where LLMs add something real
Long-horizon questions with extensive textual context and thin prediction market liquidity. A question like “Will the EU pass comprehensive AI legislation by end of 2027?” benefits from processing legislative history, committee dynamics, and procedural constraints. An LLM can synthesise more historical text faster than most researchers. If the Polymarket market is thin and sparsely traded, the LLM estimate may provide useful reference.
The caveat that cannot be overstated: LLMs have knowledge cutoffs. They cannot tell you what happened yesterday. On any question where recent information is material, an LLM estimate is anchored in the past. The Prediction Markets vs AI guide article covers where this costs traders most.
What the Research Studies Show
Studies comparing the three methods on structured question sets using Metaculus questions, Good Judgment Project archives, and Polymarket historical data from 2023-2025 show consistent patterns.
LLMs using structured prompting (base rates, counterarguments, confidence intervals) perform better than LLMs using simple direct prompting. The difference is 0.03-0.06 Brier points. Methodology matters for models just as it does for humans.
Superforecasters score 0.14-0.16 across diverse question categories. The score is stable across tournaments and time horizons above 90 days.
Liquid prediction markets on equivalent questions score 0.15-0.19. The distribution is wider than superforecasters because market quality varies with liquidity. The top quartile of liquid markets by volume matches superforecaster accuracy. The bottom quartile does not.
All three methods score 30-40% better than naive base-rate models and 50-60% better than overconfident individual expert forecasters.
The practical conclusion: on the question types where you are trading, know which method has the calibration edge. Do not assume prediction markets are always best-calibrated. Do not assume LLMs are always wrong. Map the condition to the method.
Using This as a Trader
When a liquid Polymarket market has $5 million in volume on a well-specified political question, that market price is your hardest-to-beat baseline. The question is not “what do I think” but “what do I know that the market does not, and is that genuinely information or just opinion?”
When a thin Polymarket market has $20,000 in volume on a niche regulatory question, the price is not necessarily your best reference. Superforecaster methodology applied to the same question, starting with base rates and decomposing carefully, may produce a better estimate than the crowd that assembled in that thin market. The price gap between your estimate and the market price, if verified through careful methodology, is a potential edge.
When you are researching a long-horizon question with extensive available textual evidence and the prediction market is thin, an LLM as a structured research companion helps you process historical context faster than you could alone. It does not replace the probability estimate. It helps you form it.
The Calibration guide shows how to build the personal calibration tracking system that applies superforecaster methodology to your own trading history.
Common Mistakes
Treating all prediction markets as equivalently well-calibrated is the most expensive version of this mistake. A $5 million market and a $20,000 market on nominally similar questions carry completely different information content. Apply the liquidity filter before treating any market price as an authoritative reference.
Using LLM estimates on current events without checking knowledge cutoffs. An LLM asked about an ongoing Polymarket market generates a confident-sounding probability estimate based on information that may be months out of date. The confident tone is identical whether the model is drawing on real training data or filling in plausible-sounding gaps. Check the cutoff. Verify specific claims against primary sources.
Conflating “I disagree with the market” with “the market is wrong.” A superforecaster’s edge comes from systematic methodology applied over many forecasts, not from high confidence on individual questions. One probability estimate that differs from market price is not evidence of superior calibration. Fifty tracked estimates across a specific category that demonstrate consistent accuracy is.
Frequently Asked Questions
Q: Are superforecasters better than prediction markets at forecasting? A: On long-horizon questions with thin prediction market liquidity, superforecasters may produce more calibrated estimates than the market. On liquid markets with diverse informed participants and time-sensitive information, prediction markets outperform superforecasters on speed and are roughly equivalent on long-run accuracy. The comparison depends on market liquidity and question time horizon.
Q: What does the research say about forecasting accuracy across methods? A: Research through 2025 consistently places liquid prediction markets and superforecasters at approximately the same level on long-horizon structured questions, Brier scores 0.14-0.18, with LLMs behind at 0.19-0.24. All three clearly outperform naive base rates and individual non-calibrated experts.
Q: Where do prediction markets beat human forecasters? A: Real-time information incorporation is the primary advantage. A major Polymarket market updates in minutes when news breaks. Human forecasters, even well-calibrated ones, update in hours. For any question where information from the past 24-48 hours is materially relevant, prediction markets process that information faster.
Q: Where do AI models fit in the accuracy ranking? A: Below superforecasters and liquid prediction markets on comparable question sets. The gap is largest on short-horizon questions where real-time information is decisive. It narrows on long-horizon questions with rich textual context where recent news matters less than historical pattern analysis.
Q: What is Philip Tetlock’s research and why does it matter for traders? A: Tetlock’s Good Judgment Project established through controlled research that a small group of systematically thinking forecasters consistently outperform both random chance and credentialed specialists on probability questions. The methodology those superforecasters use, base rates, decomposition, calibration tracking, rapid updating, is directly applicable to prediction market trading.
Q: What is the superforecasters Brier score compared to prediction markets? A: Superforecasters score 0.14-0.16 on structured long-horizon questions. Liquid prediction markets score 0.15-0.18 on equivalent questions. The difference is within the margin of study variation. Prediction markets have the speed advantage on information incorporation. Superforecasters have the edge on niche thin-market specialised questions.
Q: How do I use superforecaster methodology for prediction market trading? A: Start every probability estimate with the base rate for this type of event before adding any current context. Decompose the question into sub-questions with separate evidence streams. Write down your probability estimate before checking the market price. Track your estimates and outcomes by category. Update quickly when new information arrives rather than defending initial estimates.
Final Thoughts
The three-way comparison is not a tournament. It is a map.
Different methods have different advantages at different question types and time horizons. Knowing which map to use, and when to switch, is the practical skill. A trader who defaults to market prices on every question and a trader who defaults to their own model on every question are both using only one piece of the map.
The superforecaster benchmark is useful not because you will achieve Brier 0.15 immediately, but because it defines what calibrated systematic thinking looks like at scale. The habits that produce it are achievable by anyone willing to apply them consistently. Base rates. Decomposition. Honest tracking. Rapid updating.
The prediction market is measuring something real. So is a good superforecaster. So is a well-prompted LLM with appropriate constraints. Used correctly, all three together give you more than any one alone.
For the AI-specific research workflow, read AI Forecasting vs Prediction Markets.
