The Simchi-Levi Paper That Puts an Expiration Date on Human-Managed Supply Chains
Paper: “Reliability and Effectiveness of Autonomous AI Agents in Supply Chain Management”
arXiv: 2605.17036
Published: May 2026
Researchers: Carol Xuan Long, David Simchi-Levi, Feng Zhu, Huangyuan Su, Andre P. Calmon

# The Simchi-Levi Paper That Puts an Expiration Date on Human-Managed Supply Chains

The MIT Beer Game is not obscure. Every supply chain professional knows it. It is the canonical experiment taught in every MBA program — a simple four-stage supply chain where a retailer, wholesaler, distributor, and factory manage inventory with two-week lead times and stochastic demand. For 40 years, it has been the gold standard for studying the bullwhip effect: how small fluctuations in customer demand get amplified into catastrophic swings upstream.

David Simchi-Levi — one of the most cited supply chain academics in the world, the person who defined the modern understanding of supply chain analytics — just ran the Beer Game with eight different large language models instead of human players. The results will reshape every conversation about autonomous supply chain agents.

A reasoning model out of the box already beats human teams at minimizing total supply chain costs. An optimized reasoning model reduces costs by 67 percent relative to human performance. And a model post-trained with reinforcement learning — Group Relative Policy Optimization, or GRPO — cuts costs by 86 percent while reducing decision variance from 91 percent to 13 percent.

In plain English: the best autonomous AI agent manages a four-echelon supply chain for 952 dollars of total cost. Human teams cost 6,739 dollars. The human-run supply chain is seven times more expensive and seven times less predictable.

Affiliations: Harvard Business School, Massachusetts Institute of Technology, Purdue University, University of Missouri, Georgia Institute of Technology

## Why This Paper Is the Most Important Supply Chain AI Research Published This Year

The supply chain AI space is crowded with vendor claims. Every procurement technology company promises an “AI-powered” solution. Every logistics platform talks about autonomous decision-making. Until now, there has been no rigorous, independent, apples-to-apples evaluation of how these models actually perform in multi-echelon supply chains.

This paper fills that gap with authority. Simchi-Levi’s name carries weight. The Beer Game is the most studied supply chain simulation in history — every claim in this paper is measured against 40 years of human performance benchmarks. The findings are not vendor marketing. They are independent academic research conducted across multiple US universities.

The state of the world before this paper was a landscape of conflicting vendor claims and unsubstantiated ROI projections. A CPO evaluating an autonomous procurement agent had no independent framework for assessing whether the technology actually worked or whether it introduced hidden risks.

The paper changes this by providing four things no vendor has offered:

**A baseline.** Human teams running the Beer Game cost an average of 6,739 dollars with 91 percent coefficient of variation. That is the number to beat. Any AI claiming autonomous supply chain capability should be measured against this.

**A ranking.** The paper tests eight models across three tiers: non-reasoning models (GPT-4o mini costs 221 percent of human — worse than humans), reasoning models out of the box (GPT-5 mini at 58 percent, Llama 4 Maverick at 60 percent), and reasoning models with post-training (Qwen-3 4B at 14 percent — a 7x improvement over humans).

**A risk taxonomy.** The paper identifies “agent bullwhip” — decision instability generated by the LLM itself rather than by changing demand. This is a new failure mode that no supply chain executive has been trained to recognize or mitigate.

**A fix.** GRPO post-training eliminates the agent bullwhip. The coefficient of variation drops from 91 percent (humans) to 13 percent (post-trained AI). Tail events — the worst-case runs that keep supply chain executives up at night — are virtually eliminated.

## Methodology, Explained Simply

Imagine a company with four stages in its supply chain. A retailer orders from a wholesaler. The wholesaler orders from a distributor. The distributor orders from a factory. Each stage has a two-week delay between placing an order and receiving the goods. Customer demand changes unpredictably. The goal is to keep every stage stocked without holding too much inventory.

This is the MIT Beer Game. It has been run thousands of times with human players. The classic finding: humans amplify small demand changes into large order swings upstream — the bullwhip effect. Even experienced supply chain managers struggle with it.

The researchers replaced each stage with an LLM agent. Each agent could see its own inventory level and incoming orders. Each agent had to decide how much to order every week. The researchers ran 30 simulations of each configuration with identical random seeds — meaning the same demand pattern appeared in every run.

They discovered something important: not all LLMs are equal at supply chain decisions.

Non-reasoning models — GPT-4o mini — performed worse than humans. They overreacted to demand changes, ordered erratically, and incurred 221 percent of human costs. They could not be trusted to manage a supply chain under any conditions.

Reasoning models — GPT-5 mini, DeepSeek-R1, Llama 4 Maverick, Claude Sonnet 4, Gemini 2.5 Pro, Qwen-3 4B, Mistral Large — performed at or above human level out of the box. These models think step by step before ordering, and that step-by-step reasoning translates directly to better inventory decisions.

But here is the counterintuitive finding: even the best reasoning models were unreliable. Running the same model twice on the exact same demand pattern produced different orders each time. The researchers called this “agent bullwhip” — decision instability generated by the AI, not by the market.

Imagine a warehouse manager who places a different order every time you ask them the same question about the same demand data. That is what the out-of-the-box LLM agents did. And the standard fix — ask the model three times and average the answers — did not help. The variance was baked into the model’s decision policy, not into the output noise.

The fix was reinforcement learning post-training. The researchers took a base model (Qwen-3 4B) and trained it using GRPO — a method where the model learns from system-level supply chain outcomes rather than individual order decisions. The model learned that stable ordering patterns produced better total cost outcomes. It learned to ignore short-term demand fluctuations. It learned to coordinate its orders across the four echelons.

The result: cost dropped from 1,585 dollars (already 76 percent below human) to 952 dollars. Variance dropped from 26 percent to 13 percent. The worst-case run cost 1,353 dollars — compared to 11,841 for humans and 8,644 for the best out-of-the-box reasoning model.

## Results and Practical Insights

The paper’s results are not incremental. They are discontinuous — the kind of improvement that changes competitive dynamics in industries where supply chain efficiency is a profit driver.

**The post-training effect is dramatic.** GRPO post-training reduced total costs from 1,585 dollars to 952 dollars — a 40 percent improvement on top of a 76 percent improvement over humans. More importantly, it reduced the coefficient of variation from 26 percent to 13 percent. This is the number that matters for production deployment. A variance of 13 percent means the agent is predictable. A variance of 26 percent means you need buffer inventory, higher safety stock, and contingency plans.

**The tail risk argument transforms the CFO conversation.** The worst-case human-managed supply chain run cost 11,841 dollars — more than 10 times the post-trained AI’s worst case of 1,353 dollars. For a CFO evaluating an autonomous supply chain pilot, the question should not be “what does the average look like?” It should be “what does the worst case look like?” The post-trained AI’s worst case is within budget. The human worst case is an operational crisis.

**Small models with post-training outperform large models without it.** Qwen-3 4B — a 4-billion parameter model — post-trained with GRPO outperformed every out-of-the-box reasoning model tested, including GPT-5 mini and DeepSeek-R1. This is the same pattern the Silicon Valley Certification Hub Chief AI Officer research has identified across multiple domains: architecture and training matter more than raw model size. A small model with the right training process beats a large model without it.

**Agent bullwhip is a real operational risk that requires a policy-level fix.** The paper demonstrates that repeated sampling — asking the model three times and averaging — does not fix decision instability. The instability is at the policy level. The model’s decision policy is inherently stochastic, and averaging outputs does not change the underlying policy. Only retraining the policy itself — through GRPO post-training — addresses the root cause.

## Key Takeaways for Supply Chain and Procurement Leaders

**Start piloting autonomous AI agents now, but require post-training before production deployment.** Out-of-the-box reasoning models are impressive in the lab. They will beat your human teams at cost minimization in a controlled simulation. But the variance will cause real-world problems — inventory mismatches, stockouts, emergency shipment costs — that the simulation does not fully capture. Require RL post-training as a gating criterion for any vendor claiming autonomous supply chain capability.

**Request the vendor’s agent bullwhip measurement, not just their average cost claim.** Any vendor can run a simulation and report average cost improvement. Very few can measure and report the coefficient of variation of their agent’s decisions across repeated runs. If a vendor cannot tell you the variance of their agent’s order decisions under identical demand conditions, they have not tested for reliability. The paper clearly shows that average performance and worst-case performance are different metrics.

**The CPO role is about to get a new technical competency requirement.** Understanding the difference between out-of-the-box LLM performance and post-trained RL performance will be a basic skill for supply chain leaders. The paper provides a clear framework: model selection, policy constraints and guardrails, centralized data sharing, and prompt engineering are the four levers. Post-training is a fifth lever that compound-returns on the others.

**Watch the small model + RL training pattern.** The finding that a 4-billion parameter model with GRPO post-training outperforms every larger model out of the box has implications for deployment cost and latency. Smaller models are cheaper to run, faster to respond, and easier to deploy on edge infrastructure. If the training process — not the model size — is the differentiator, then companies with strong in-house AI training capability have an advantage over companies relying on vendor-provided base models.

**Do not let the 86 percent cost improvement distract from the risk management framework.** The headline is dramatic. The paper will generate board-level attention. The responsible Chief AI Officer response is to pair the cost improvement conversation with a reliability and risk conversation. The technology works. It also introduces a failure mode — agent bullwhip — that no existing supply chain risk management framework addresses. The first company to build agent bullwhip monitoring into their supply chain operations will have a structural advantage over competitors who deploy AI without reliability safeguards.

## Thanks to All Authors

Carol Xuan Long — Harvard Business School, Cambridge, MA

David Simchi-Levi — Massachusetts Institute of Technology, Cambridge, MA

Feng Zhu — Purdue University, West Lafayette, IN

Huangyuan Su — University of Missouri, Columbia, MO

Andre P. Calmon — Georgia Institute of Technology, Atlanta, GA

Want to know how this applies to your company?
At Silicon Valley Certification Hub, we help you align AI + Strategy. Our team works directly with your directors and teams to assess AI readiness, identify gaps, and build a clear path forward — tailored to your business context.
Book a time with our CEO, Alejandro Cuauhtemoc-Mejia:
https://calendar.app.google/2ihQf2JH3D9uJBe68
Silicon Valley Certification Hub
3000 El Camino Real, Building 4, Palo Alto, CA