NL2SQL Is for Exploration, Not Production

When you're exploring, not reporting. If someone is asking a question no one anticipated, a human is reviewing the output, the analysis is throwaway, and an approximate answer is good enough, NL2SQL is a reasonable and genuinely useful tool — it can reach questions no template covers. The line to watch is whether the answer has to be right every time. The moment a number feeds a dashboard, a KPI, a customer-facing product, or a decision someone will defend, you've crossed from exploration into production — and that's where verified, templatized queries belong.

It's improving on benchmarks, but the failure modes are structural, not just a matter of model quality. Probabilistic generation means non-determinism no matter how good the model gets, and the worst errors are the silent ones — semantically wrong SQL that runs cleanly. Even at 90%+ benchmark accuracy on tidy schemas, a 1-in-10 silent error rate on production analytics is unacceptable for anything you'd report or act on. Better models reduce the rate; they don't change the architecture's risk profile.

Those help with safety and cost control — preventing destructive statements, capping runaway scans — but they don't make the answer correct. A read-only sandbox happily runs a query that joins the wrong tables. Validation can catch malformed SQL, not a wrong-but-valid aggregation. The only reliable way to guarantee correctness is to verify the query logic itself ahead of time, which is exactly what a template is.

It narrows the surface deliberately, and that's the trade: verified-but-curated instead of flexible-but-unreliable. For the questions that matter — your real KPIs and metrics — you want the curated set. For genuinely open-ended exploration, an LLM is a reasonable tool, as long as everyone understands the output is a draft to be checked, not a number to be trusted. Most production analytics needs are a known, finite set of questions, which is precisely what templates cover well.

In front of the templates, not in place of them. The LLM is excellent at understanding a question phrased in plain English, classifying intent, choosing the right verified template, and extracting the parameters to pass into it at runtime. That gives you the natural-language experience people want and the deterministic, auditable execution your business needs — the model selects and parameterizes, the verified query does the rest.

Each analytic is a verified, parameterized query exposed as a described API. Because the description travels with the tool, an MCP-connected agent can read the full menu of analytics available to it — what each one answers and which parameters it accepts — and simply select and call the right one, passing arguments at runtime. There's no real-time SQL authoring and no query planning: the agent identifies the analytic and the verified query runs on a known route, which is faster. And because the response is a compact, pre-shaped insight rather than raw rows, the payload sent back into the model's context is much smaller — which directly lowers latency and token cost in agentic and chatbot use cases.

Benchmark schemas are clean, well-documented, and designed to be answerable. Production warehouses have ambiguous column names, overlapping tables, undocumented business rules, soft-deletes, and fiscal logic that isn't in the schema at all. Accuracy that looks strong on a benchmark drops substantially against that reality — and the gap is widest on exactly the multi-join, edge-case-heavy queries that carry the most business value.

If your schema is genuinely a couple of clean tables, NL2SQL will be more reliable than average. But schemas rarely stay simple: they accumulate joins, history, exceptions, and special cases as the business grows. The value of templatizing shows up precisely as complexity increases, so building on verified queries early means you don't have to re-architect the moment your data — and your risk — gets real.