Source<\/p>\n
“Using LLMs for Market Research”<\/strong> \u2014 Harvard Business School Working Paper 23-062. James Brand, Ayelet Israeli, Donald Ngwe. The study queried large language models dozens of times per survey question to build a distribution of simulated consumer responses, then compared those distributions against real human willingness-to-pay data.<\/p>\n<\/div>\n The researchers found baseline LLMs could reasonably approximate willingness-to-pay for features that already existed in a product category. Ask the model about a feature customers have been buying for years, and it does an okay job guessing what they’ll pay for it.<\/p>\n New features are where it fell apart. Ask the same model to predict demand for something genuinely novel, a feature with no real-world pricing history for it to pattern-match against, and the estimates got shaky. In some cases the direction flipped entirely, the model said “customers will pay more” when they would actually pay less.<\/p>\n The fix the professors found wasnt more prompting cleverness. It was fine-tuning the model on prior human survey data from comparable products and customer groups. Once grounded in real human responses, accuracy improved. Without that grounding, the model was guessing with confidence, which is the most dangerous kind of guessing.<\/p>\n Market research is just the domain where someone bothered to run the controlled test. The underlying assumption the professors were probing is much bigger: that an LLM’s output on a question of human preference or judgment can stand in for actually asking humans.<\/p>\n That assumption shows up everywhere executives make calls today. Pricing decisions. Positioning a new product line. Reading “signal” out of a chatbot transcript about whether a hire is a culture fit. Ranking which market to expand into first. None of those are market research surveys, but all of them lean on the same premise the HBS paper just stress-tested and found breaks under novelty.<\/p>\n Some willingness-to-pay estimates from the models werent just inaccurate. They pointed in the wrong direction entirely.<\/p><\/blockquote>\n A pricing team that fine-tunes a model on five years of real customer data, then asks it to forecast demand for a feature adjacent to what it already knows, is doing something close to what the HBS paper validated works. A team that opens a chat window, describes a brand-new product concept, and takes the model’s confident answer as market signal is doing the thing the paper found fails, and fails silently. No error message. No confidence interval. Just a wrong number that looks exactly like a right one.<\/p>\n Thats the gap most companies havent mapped yet: which of their AI-assisted decisions are grounded in real prior data, and which ones are just a model pattern-matching on vibes and calling it judgment. Figuring out that distinction across pricing, hiring, and strategy, not just marketing surveys, is exactly what an AI Assessment for companies<\/a> is built to surface.<\/p>\n The professors’ recommendation is blunt: AI should augment human research, not replace it. Every executive currently treating a model’s output as a finished answer instead of a first draft is running the experiment the HBS paper already ran. They just dont have the paper’s control group.<\/p>\n Key conclusions:<\/p>\n My sharpest interpretation: LLMs do not eliminate the need for market research. They make existing market research more reusable.<\/strong> Their value increases when a company already owns relevant, high-quality customer data.<\/p>\n One important caveat: this is an April 2026 working-paper revision, but most principal experiments used GPT-3.5 Turbo in 2024. The precise performance of today\u2019s models may differ; the paper\u2019s deeper warning about validation and generalization remains highly relevant. Read the HBS working paper<\/a>.<\/p>\n It means every AI-generated “insight” feeding a strategic decision needs a documented data lineage. If a model’s answer isnt grounded in real prior data comparable to the question being asked, a Chief AI Officer should treat it as a hypothesis, not a finding, and require a human validation step before it reaches a decision memo.<\/p>\n<\/div>\n Because LLMs generate answers by pattern-matching against training data, and truly new features have no real-world pricing history to pattern-match against. Without comparable prior data, the model produces a confident-sounding number that has no reliable relationship to actual human preference.<\/p>\n<\/div>\n An AI Assessment for companies maps exactly which business decisions are grounded in real prior data versus which ones rely on unverified model output, the same distinction the HBS paper found separates accurate predictions from wrong-signed ones. Silicon Valley Certification Hub built its assessment process around that gap.<\/p>\n<\/div>\n The output can look just as credible when its wrong as when its right, with no built-in flag for low confidence. A pricing or positioning call made on a wrong-signed estimate can point a launch in exactly the wrong direction before anyone catches it.<\/p>\n<\/div>\n Separate AI-assisted decisions into two buckets: ones grounded in fine-tuned models with real comparable data, and ones based on raw model judgment about something genuinely new. The second bucket needs a human research step before it becomes a strategic decision, not after.<\/p>\n<\/div>\n<\/div>\n <\/p>\n![\"Silicon](\"https:\/\/svch.io\/wp-content\/uploads\/2026\/07\/silicon-valley-certification-hub-alejandro-cuauhtemoc-mejia-llm-wrong-signed-predictions-consumer-preference-body1.png\")
What the model got right, and exactly where it broke<\/h2>\n
This isnt really a market research story<\/h2>\n
![\"Silicon](\"https:\/\/svch.io\/wp-content\/uploads\/2026\/07\/silicon-valley-certification-hub-alejandro-cuauhtemoc-mejia-ai-judgment-executive-decision-making-body2.png\")
Where this actually bites a company<\/h2>\n
\n
Frequently Asked Questions<\/h2>\n
What does this mean for a Chief AI Officer?<\/h3>\n
Why did the Harvard researchers get “wrong-signed” results specifically on new features?<\/h3>\n
How does an AI Assessment for companies relate to this research?<\/h3>\n
Whats the risk of treating an LLM’s output as finished market research?<\/h3>\n
What should executives do differently starting now?<\/h3>\n