<\/p>\n
<\/p>\n
Task Success on Real Enterprise Analytics Queries<\/p>\n
Tested at two Fortune 500 companies. The difference is not a better model \u2014 it is a fundamentally different architecture.<\/p>\n<\/div>\n
<\/p>\n
For roughly a decade, a simple promise has driven enterprise AI investment: let non-technical teams ask questions in plain English and get answers from company data. Type “show me sales by region this quarter” and the AI writes the SQL, hits the database, and returns a chart.<\/p>\n
It has not worked. Not in any serious enterprise deployment. And the reason is not that the AI cannot write SQL. The reason is that enterprises do not give AI systems direct database access.<\/p>\n
A new paper from Dialpad \u2014 the company behind the contact-center AI platform \u2014 documents the failure and the fix at two Fortune 500 companies. For every Chief Data Officer, Chief Information Officer, and Head of Business Intelligence who has watched a Text-to-SQL pilot fail, this paper is the post-mortem and the blueprint for what comes next.<\/p>\n
The Text-to-SQL gap has nothing to do with SQL accuracy. Academic benchmarks show 80%+ accuracy. The gap between 80% on a benchmark and 18% in a real enterprise comes down to three structural problems:<\/p>\n
Fragmented, Moving Data<\/p>\n
Enterprise data lives across warehouses, lakes, and engines. Schema changes constantly. Business logic lives in views and application code \u2014 not raw tables. Benchmark queries break immediately against a moving production system.<\/p>\n<\/p>\n<\/div>\n<\/div>\n
Governed Permissions<\/p>\n
Different teams see different data. Some metrics are confidential. A Text-to-SQL system with direct database access bypasses every access control. One bad prompt becomes a compliance incident.<\/p>\n<\/p>\n<\/div>\n<\/div>\n
Business Logic Lives in APIs, Not Tables<\/p>\n
The correct way to get “average daily call volume for the support team” is not raw SQL \u2014 it is calling the analytics API endpoint that already computes that metric with the correct business logic applied.<\/p>\n<\/p>\n<\/div>\n<\/div>\n<\/div>\n
<\/p>\n
Think of a corporate building with a guard at every door. Traditional Text-to-SQL tries to give every visitor a master key \u2014 direct database access. Dialpad’s Analytic Agent takes a different approach: no master key. Instead, the agent understands which rooms exist, which doors lead where, and how to ask the guard for permission.<\/p>\n
Five specialized LLM agents work together across six stages:<\/p>\n
<\/p>\n
<\/p>\n
| Model Tier<\/th>\n | Execution Success<\/th>\n | Relative Cost<\/th>\n<\/tr>\n<\/thead>\n |
|---|---|---|
| Gemini 2.5 Pro (most powerful)<\/span><\/td>\n | 96.67%<\/span><\/td>\n | $$$<\/td>\n<\/tr>\n |
| Gemini 2.5 Flash (production choice)<\/span><\/td>\n | 94.44%<\/span><\/td>\n | $ \u2713<\/td>\n<\/tr>\n |
| Gemini 2.5 Flash-Lite (smallest)<\/span><\/td>\n | 44.44%<\/span><\/td>\n | \u00a2<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n The gap between Pro and Flash: 2.2 percentage points. The gap between Flash and Flash-Lite: 50 percentage points.<\/strong> The threshold model capability is real but not high. A sufficiently capable model with the right architecture beats a powerful model with the wrong one. Enterprise analytics AI is an orchestration problem, not a benchmark problem.<\/p>\n <\/p>\n <\/p>\n Core Architectural Insight<\/p>\n The business logic lives in the APIs. The permissions live in the governance layer. The LLM navigates them. Governance is not a constraint the AI must work around \u2014 it is the reason the AI works at all.<\/strong><\/p>\n<\/div>\n <\/p>\n Stop investing in Text-to-SQL. Start investing in API orchestration.<\/p>\n The paper makes the empirical case. Direct database access through LLMs fails in enterprise settings. If your data platform team is building Text-to-SQL features, this is the evidence you need to redirect investment.<\/p>\n<\/p>\n<\/div>\n<\/div>\n Build the governance layer first.<\/p>\n The paper’s architecture puts permissions, entity resolution, and validation between the LLM and the data. The governance layer is not an add-on \u2014 it is the foundation.<\/p>\n<\/p>\n<\/div>\n<\/div>\n Do not optimize for the most powerful model.<\/p>\n Flash delivered 94.44% execution success at a fraction of Pro’s cost. The architecture \u2014 governed APIs, retry logic, deterministic date handling, caching \u2014 compensates for model capability differences. Optimize for architecture and cost, not benchmark scores.<\/p>\n<\/p>\n<\/div>\n<\/div>\n The retry loop is the difference between a demo and a deployment.<\/p>\n Any analytics AI can handle the happy path. Production systems handle the unhappy path gracefully. The paper’s two-tier retry \u2014 programmatic for predictable errors, LLM-powered for ambiguous ones \u2014 is the production pattern.<\/p>\n<\/p>\n<\/div>\n<\/div>\n Caching is the unsung economics story.<\/p>\n 64% cost reduction and 22% latency reduction from caching. In enterprise analytics, many queries are repeated \u2014 same metric, same team, same time range. Caching captures these and eliminates re-execution cost entirely.<\/p>\n<\/p>\n<\/div>\n<\/div>\n The role of the Chief Data Officer is shifting.<\/p>\n Instead of writing SQL for every business user request, the data team builds analytics APIs with clear interfaces, documented metrics, and governance rules. The AI navigates them. The team maintains them. The architecture scales.<\/p>\n<\/p>\n<\/div>\n<\/div>\n<\/div>\n <\/p>\n \n Gundeep Singh \u2014 Dialpad Inc., San Francisco, CA Your Text-to-SQL pilots are failing not because AI models are weak, but because enterprise governance prevents direct database access\u2014and that’s actually correct security policy. The 92% success rate from governed API orchestration shows you can solve this by building a different architecture layer that sits between AI systems and your data, rather than trying to make AI safer at writing SQL. This shifts your investment from model improvement to orchestration infrastructure.<\/p>\n<\/div>\n Benchmarks test whether AI can write correct SQL syntax in isolation, but enterprise data environments involve access controls, schema complexity, business logic rules, and governance policies that academic datasets don’t replicate. When an AI system hits a permission denial or schema constraint, it fails the entire query\u2014not because the SQL was wrong, but because it attempted something the system architecture forbids. The Dialpad research shows that wrapping multiple smaller API calls through a governance layer succeeds where single-query SQL approaches break.<\/p>\n<\/div>\n Companies evaluating their AI analytics capabilities through an AI Assessment framework should distinguish between model capability (what the AI can theoretically do) and system capability (what the architecture actually allows). Silicon Valley Certification Hub recommends using this paper’s findings to audit whether your organization is testing Text-to-SQL in a realistic enterprise context with actual security policies, not in a sandbox\u2014because the gap between those two scenarios is where billions in failed AI investments have landed. A proper assessment should measure orchestration readiness, not just model performance.<\/p>\n<\/div>\n Start by mapping your governance requirements and existing API ecosystem before selecting a query approach\u2014the architecture choice must follow your security model, not precede it. Then pilot governed API orchestration with a small set of common analytics queries rather than attempting unrestricted natural language SQL, because the research shows this delivers immediate results while remaining compliant with your data access policies. This approach shifts your analytics AI from a long-term research bet to a near-term operational win.<\/p>\n<\/div>\n<\/div>\n <\/p>\n |