Definitions Beat Model Choice for Data-Analysis Agents

The Top of Every Text-to-SQL Leaderboard Is an Agent System, Not a Model

Ask which model is best at data analysis and the leaderboards give you a name that is not a model. On Spider 2.0-Snow the top entry, at 96.70, is Genloop's Sentinel Agent v2 Pro. On Spider 2.0-Lite it is the Tianqiong Data Agent running GLM 5.2, at 76.23. On Spider 2.0-DBT it is SignalPilot's agent, at 65.6. On BIRD's test set it is AskData from AT&T, at 83.40. Each is a vendor's pipeline: retrieval, planning, sampling, voting, repair loops. The base model is one part, and often the page does not say which one.

That matters because the question a team asks is different. You are not buying a leaderboard entry. You are choosing a model and a design for your own warehouse, with your own table names, your own odd columns and your own meaning of "sale". On that question the public numbers say less than they seem to, and the best evidence points somewhere else: at the written context you give the model, not the model.

This piece dates every number, says where each comes from, and ends with how to run the one test that counts: your own questions.

The Benchmarks, Dated

Every score below was read from a primary page or paper on 2026-11-13 unless the row says otherwise. Where a page gives no date, the row says so.

We could not load the official DABstep page, so the DABstep row uses an aggregator's copy of the leaderboard. The Spider 2.0 page states no update date for any entry.

Two things stand out before any argument. Scores on the three public Spider settings do not agree with each other, so "Spider 2.0 accuracy" is not one number. And the BEAVER row is the only one measured on data a model could not have seen.

Two of the Big Benchmarks Are Too Noisy to Rank By

A leaderboard works only if the answer key is right and the test does not move. Both assumptions have taken damage this year.

Spider 2.0-Snow has errors in its answer key. A January 2026 audit (arXiv 2601.08778, later in PVLDB) reviewed the tasks with public gold SQL: 121 of the 547. It reports an annotation error rate of 62.8% on Spider 2.0-Snow and 52.8% on BIRD Mini-Dev. When the authors corrected a subset of BIRD's dev set and re-scored 16 open-source agents, relative scores moved from about minus 7% to plus 31%, and rankings shifted by up to nine places. Rankings on the corrected subset correlated only weakly with the original order (Spearman 0.32).

Separately, Emergence AI examined 32 Spider2-Snow mismatches and attributed 28 to the benchmark itself: 20 to live data that changed after the gold answers were fixed, 5 to differences between BigQuery and Snowflake, 3 to revoked data shares. Only 4 were wrong query logic. They propose a trimmed set of 524 tasks.

DABstep is saturated. The launch paper reported the best agent at 14.55% on the hard tasks. The leaderboard copy we could reach, updated Aug 26, 2026, has sixteen entries at 100% on hard, several submitted under names like "test". A test where sixteen systems tie at the ceiling ranks nothing.

This is the pattern Donella Meadows describes for any target. People tune to what is scored.

”

"If the quality of education is measured by performance on standardized tests, the system will produce performance on standardized tests." — Donella H. Meadows, Thinking in Systems (§ "Seeking the Wrong Goal")

Even the model-ranking firms now hedge against it. Artificial Analysis rebuilt its Intelligence Index on Oct 28, 2026 so that 60% of it rests on private test sets or held-out answers. See also the contamination problem and how to write evals that can be climbed without being gamed.

Private Warehouses Score a Fraction of Public Ones

BEAVER is the clean test, because its three warehouses are private (an academic IT department's Oracle warehouse and two MySQL systems at a research lab and a residential facility). Its tables average 101.5 per database against BIRD's 6.8. The paper's v3, dated May 13, 2026, reports end-to-end execution accuracy for seven models and four methods. The best is ReFoRCE, an agent method, at 11.4% with Claude-4.5-Sonnet and 10.8% with GPT-5.2. The same method and model reach 62.9% on Spider 2.0.

The first version of the paper, in 2024, reported close to zero for GPT-4o with retrieved tables. Models improved; the gap did not close. Giving the agents oracle hints for every subtask lifts accuracy to 25.9% in the paper's table (the abstract says 30.1%; the two disagree).

Teams that publish results on their own warehouses report a wide range.

• LinkedIn SQL Bot (July 2025, 300+ weekly users): 53% of responses correct or close to correct, by expert review on an internal set.
• Snowflake Cortex Analyst (Aug 2024, vendor, 150 internal questions): 90%+ for the product with a semantic model, 51% for single-prompt GPT-4o. This is Snowflake grading Snowflake, on old models, and nobody has replicated it.
• Uber QueryGPT: published no accuracy figure. It says run-to-run swings near 5% should not drive decisions, and that invented tables and columns remain unsolved.

Huyen's summary of public benchmarks fits all of this.

”

"Public benchmarks can help you weed out bad models, but they won’t help you find the best models for your applications." — Chip Huyen, AI Engineering (p. 410)

The Biggest Lever in the Evidence Is Written Context

Three studies vary the context and the model, and the context wins each time.

Rumiantsau and Fokeev (arXiv 2604.25149, April 28, 2026). 100 questions on the Contoso retail dataset in ClickHouse, three models: Claude Opus 4.7, Claude Sonnet 4.6 and GPT-5.4. With schema only, accuracy was 45.5% to 50.5%. With the schema plus a 4 KB hand-written markdown file of measures, conventions and disambiguation rules, it was 67.7% to 68.7%, a gain of 17 to 23 points, significant at p below 0.01. Within each condition the three models were statistically indistinguishable. The authors attribute essentially all the significant variance to the document, none to model choice within the tier.

dbt Labs (April 7, 2026). On the ACME Insurance benchmark, 11 questions run 20 times each, the team added three dbt models, written by an LLM with no hand-written code, and re-ran. Free text-to-SQL went to 90.0% on Sonnet 4.6 and 84.1% on GPT-5.3 Codex. Through the semantic layer, the same models scored 98.2% and 100%. In their reasoning sweep, GPT models on the highest setting took over 20 seconds per query against 8 on high, "with no meaningful accuracy improvement", and Sonnet 4.6 outscored Opus 4.6 on the semantic layer. The authors sell dbt, the set is small, and the free-SQL arm loaded the whole schema as context, which they say does not scale. Read the direction, not the decimals.

Tribal knowledge. Agarwal et al. (Feb 2026) bolt a store of learned facts about the data onto any text-to-SQL agent and report gains of up to 16.9% on Spider 2.0 and 13.7% on BIRD. A COLM workshop paper, Gwimm and Eisenach (Aug 2026), compared tuning context artifacts with tuning the retrieval harness on 5,176 retail queries: about 12 to 25% similarity gains for the artifacts, 3 to 12% for the harness. On BEAVER the best variant reached 9.00% against 6.33%, which the authors report is not significant (p = 0.12). The direction agrees; the size does not hold up everywhere.

Vendors say the same thing in their own docs. Databricks writes: "Defining business logic as SQL expressions produces more consistent results than text instructions, because Genie applies the logic exactly as written rather than interpreting it from natural language" (Tune Genie Agent quality, page dated Nov 5, 2026.) Write the measure once, as code, and the model stops guessing what "win rate" means. See the Databricks stack for where this sits in the platform.

Hubbard states the principle in measurement terms.

”

"Business managers need to realize that some things seem intangible only because they just haven’t defined what they are talking about. Figure out what you mean and you are halfway to measuring it." — Douglas W. Hubbard, How to Measure Anything (§ "Clarification Chain")

The context work is mostly this. It is not prompt tuning. It is deciding what "active customer" and "sale" mean, writing it down, and pointing the model at it. That is also the argument for report tools over open SQL when a question has a fixed meaning.

Free SQL Fails Quietly, a Governed Layer Fails Loudly

Accuracy gaps hide a second difference: what a wrong answer looks like.

dbt's first run is the clearest data. Before any remodelling, over all 11 questions, free SQL scored 64.5% on both Sonnet 4.6 and GPT-5.3 Codex. The semantic layer scored 72.7%. The semantic layer was perfect on the questions it could express (100.0%) and scored 0% on the ones that needed more entity hops than it supported. Its failure was an error message. The authors' own words: with text-to-SQL, "failure looks like a plausible but incorrect answer"; with the semantic layer, "failure looks like an error message."

Our own small run shows the quiet kind. An analyst agent at MMNTM answers questions against a private Postgres warehouse by writing free SQL. On 29 known-answer questions (one skipped by the run window, counted as a miss), gpt-6.1-sol got 25 right and gpt-6-luna got 22 (Nov 11, 2026). The misses were nearly all about meaning, not syntax: which teams count toward a total, and which of two sale counts the question wanted. Both queries ran. Both returned a number. The number was wrong in a way no SQL linter would catch.

That is one run on one warehouse, and we use it as an illustration only. It does fit what the larger studies find: the model wrote valid SQL, and the missing piece was a definition.

Pick the Model Tier First, Then Stop

Which models count as current matters less than it used to, but you should name them. As of Nov 13, 2026, per Artificial Analysis coverage: Claude Opus 5.5 led its Intelligence Index v5 at 58 when the index launched on Oct 28. Claude Sonnet 5.5 and GPT-6.1 Sol sit just above Gemini 4 Hydrogen, which scored 54 on Nov 12 and costs a third of Sonnet's price but about 2.5 times Sol's.

No published text-to-SQL study we found tests that generation. The newest model comparisons above used Opus 4.7, Sonnet 4.6, GPT-5.4, GPT-5.3 Codex and GPT-5.2: one to two generations old. Where they tested several models in a tier, the models tied or traded places, and the larger or slower setting bought nothing. Treat that as a prior, not a law.

The first-party run adds one data point on cost. Per correct answer, gpt-6.1-sol cost $0.062 ($1.56 for the run) and gpt-6-luna cost $0.0042 ($0.09): about 15 times less for three fewer right answers. With 29 questions the 95% intervals are wide, roughly 69% to 94% for 25 of 29 and 58% to 88% for 22 of 29. They overlap. The cheaper model is not shown to be worse. See how to budget agents by cost per correct answer.

So the order of work is:

1. Choose a tier on cost and latency. A mid-tier current model is a safe start.
2. Write definitions and curate tables until the eval stops improving.
3. Re-test one tier up and one down. Move only if your own questions say so.

Run Your Own Eval: About 30 Known-Answer Questions

The public numbers cannot tell you how a model does on your warehouse. Thirty questions can, to a first approximation. When you start knowing almost nothing, a little data buys a lot.

You start knowing almost nothing about how your agent will do, so thirty graded questions are worth a lot. They will not resolve a three-point difference, so do not try to.

Two traps. Once the agent's builders see the questions, they will tune to them, so hold out a third you never look at; the hill-climbing article covers how. And an eval that never changes goes stale, because your business definitions will. See building agent evals for the wider method, and RAG reality check for why a bigger retrieval index is not the same as better context.

What to Do Monday

• Ignore public tops when choosing. Use BEAVER-style gaps as your prior: expect a fraction of the public score on your warehouse.
• Write the definitions file. Measures, filters, grain, "as of" rules, and a list of known traps. Start at 4 KB, the size that gave Rumiantsau and Fokeev 17 to 23 points.
• Route fixed-meaning questions to governed tools. Let free SQL handle the rest, labelled unverified, with its assumptions stated.
• Build the 30-question eval this week. Independent answers, per-check grading, a held-out third.
• Choose a model tier last, and test one tier down. The cheap model may tie.