Your AI Forecaster Can Predict Markets. It Cannot Predict People.

Here is what the frontier AI models can do with remarkable skill: forecast GDP growth, predict quarterly sales trends, estimate the probability of an interest rate change by month-end.

Here is what they cannot do: tell you whether a competitor’s CEO will follow through on a stated strategy. Judge whether a regulator actually intends to enforce a new rule. Model how your own organization’s decision-making process will shift over the next quarter.

The difference is not one of degree. It is a systematic failure — and new research proves it.

A team of 81 researchers across multiple institutions has built the largest open forecasting benchmark ever constructed. Bench to the Future 2 (BTF-2) contains 1,417 pastcasting questions — forecasts about known outcomes asked as if unknown — backed by a frozen 15-million-document research corpus. Every agent’s reasoning traces are captured. The benchmark detects accuracy differences as small as 0.004 Brier score.

The hybrid advantage: A multi-agent system outperformed every single frontier model by 0.011 Brier. The differentiator: pre-mortem analysis and black swan consideration — the AI examining its own blind spots.

The blind spots — validated by expert human forecasters:

• G1: Human Incentive Assessment — AI cannot model why leaders really act
• G2: Follow-Through Likelihood — AI cannot judge whether stated plans will happen
• G3: Institutional Process Modeling — AI cannot model how organizations make decisions

Your AI forecaster can tell you the most likely market direction. It cannot tell you whether your board will approve the pivot. That is not a bug to be patched. It is a fundamental limit on what AI can do for strategic decision-making.

Executive Summary

The benchmark: 1,417 pastcasting questions, 15M documents, 0.004 Brier sensitivity, full reasoning trace capture — the most rigorous evaluation of AI strategic reasoning ever conducted.

The hybrid advantage: Multi-agent forecaster outperforms every single frontier model by 0.011 Brier through pre-mortem analysis and black swan consideration.

The three blind spots — validated by expert human forecasters:

• G1 — Human incentive assessment: Why leaders actually act, beyond stated rationale
• G2 — Follow-through likelihood: Whether stated plans will be executed as announced
• G3 — Institutional process modeling: How organizations actually make decisions

Paper at a Glance

The Benchmark That Changes How We Evaluate AI Forecasters

BTF-2 is structurally different from previous benchmarks in ways that matter for executives who depend on forecasts.

Pastcasting methodology eliminates hindsight bias. Agents predict known outcomes against a frozen document corpus containing no outcome information. The agent cannot cheat — it must reason from the same information a human forecaster had at the time.

15 million documents ensure depth without contamination. Every agent searches the same dataset. If Agent A beats Agent B, you know it was better reasoning, not better data.

Full reasoning trace capture means evaluators see why an agent made its prediction — enabling expert human forecasters to identify the three blind spots. Without traces, the failures would remain invisible.

0.004 Brier sensitivity detects improvements lost in noise with less precise tools. This granularity enables optimization that compounds across thousands of forecasts.

The Three Blind Spots

Why Business Leaders Should Care

Every strategic forecast contains a human element. Market forecasts depend on what regulators will do. Competitive forecasts depend on what rivals will decide. Organizational forecasts depend on execution.

The three blind spots have been validated by expert human forecasters across thousands of predictions. They are not edge cases. They are the central weakness of current AI forecasting.

The Hybrid Forecasting Advantage

The paper’s other major finding — multi-agent systems outperform any single model — has a clear business implication: do not rely on one AI forecasting tool.

The 0.011 Brier improvement may sound small. In forecasting, it is not. The difference between first and second place in competitive tournaments is often less than 0.01 Brier. Across thousands of organizational forecasts, this compounds into materially better strategic decisions.

Practical implication: Organizations building forecasting pipelines should deploy multi-agent ensembles, not single models. The infrastructure cost is higher. The accuracy payoff is proven.

What Business Leaders Should Do Next

1. Segment your forecasts — Categorize every strategic forecast by whether it depends on data trends or human behavior.
2. Audit your AI forecasting tools — For each tool, assess performance on the three blind spots.
3. Deploy multi-agent ensembles — Replace single-model forecasting with hybrid systems.
4. Require pre-mortem analysis — Before every AI prediction, identify what the model might be missing.
5. Adopt Brier score — Standardize forecasting accuracy measurement across AI and human teams.
6. Train your teams — Help strategy and risk teams understand the three blind spots.
7. Build human-in-the-loop processes — For behavior-dependent forecasts, AI provides input; humans provide judgment.

Conclusion