Why AI (LLMs) Still Fail at Data Analytics.
A frontier model with unlimited tokens scored 21% on Anthropic's own analytics evals. The model was never the bottleneck — the missing piece is a written-down definition of what the business means by its own words.
Twenty-one percent. Sit with that number.
This wasn't a toy model on a rigged benchmark — a frontier model, effectively unlimited context, pointed at a well-run data warehouse, answering the kind of analytics question a real business actually asks. Twenty-one percent.
The model wasn't the bottleneck. What was missing is something no amount of intelligence can supply from the inside: a written-down answer to what the organization means by its own words.
HOW MANY USERS?
Three teams. Three governed queries. Three correct answers — and the database never told anyone which one was meant.
All three numbers are correct. No bad query, no mistake. The database knows exactly what a user_id is. It doesn't know which of the three organizational definitions of "user" the person asking actually meant.
A human analyst resolves this by asking a follow-up question. An AI agent, left to itself, usually doesn't. It picks an interpretation, writes flawless SQL against it, and hands you a number that looks exactly as confident as the right answer would.
LLMs can write the SQL. They still don't know what the number means.
This isn't an AI problem. SQL and dashboards have disagreed with each other for as long as organizations have had more than one of them. Finance and growth dashboards have reported two different "active user" counts since long before any model was writing queries. AI changes the speed, the scale, and how easy the disagreement is to miss. A confidently wrong number from an agent is a data-governance problem that predates the agent. Same problem. New interface.
- This predates AI. SQL, dashboards, and spreadsheets have been giving organizations conflicting numbers for years — AI just runs into the same ambiguity faster and at greater scale.
- SQL correctness, analytical correctness, and business correctness are three different things. A query — human-written or AI-generated — can clear the first two and still fail the third.
- This usually isn't hallucination. The model isn't inventing a table — it's making a plausible, defensible interpretation of an ambiguous business term.
- Anthropic's own data team reported accuracy topping out around 21% without a governed source of truth, and above 95% once one was in place. These are their internal, reported figures, not an independently reproduced benchmark.
- The fix isn't a better prompt, and a semantic layer is one implementation of it, not the whole answer. What matters is whether a governed source of truth exists for the model to retrieve at all.
- It decays. Anthropic reported accuracy drifting from ~95% back down to ~65% within a month once definitions stopped keeping pace with the business.
The query can be perfect and still be wrong
Split "correct" into three separate claims. An LLM can satisfy the first two and still fail the third without anything looking broken.
Three levels of correctness
A query that clears levels one and two looks, by every technical signal available, like a success. It executed. It returned rows. Nothing threw an error. The only way to know it's wrong is to already know what the business meant — which is exactly the information the query itself can't tell you it's missing.
Not AI. SQL. Dashboards.
Remove the AI agent from the story entirely and the problem remains. Before anyone queried a model, an analyst wrote a SQL query defining "active user" one way. A different team built a dashboard defining it another way. A third team built a spreadsheet with a third definition, because nobody told them the first two existed. Nobody was wrong. Nobody made a mistake. Almost every organization with more than one data team already has this problem.
The same disagreement, no AI involved
Dashboard A
"Active" = logged in within 30 days
Dashboard B
"Active" = subscription status is current
A SQL query
"Active" = a third, unwritten business rule
Most "AI analytics" work actually is schemas, permissions, table ownership, keeping a data model coherent as the business shifts underneath it. That's traditionally been database administration — unglamorous, rarely the headline. The hardest part of reliable AI analytics isn't model intelligence; it's the unglamorous work of keeping data, definitions, and sources of truth coherent. DBAs and data engineers have been doing that work all along.
This isn't specific to any one vendor's model. Any system — a person, a dashboard, or an AI agent — asked to answer a business question against a warehouse with no governed source of truth will hit some version of this problem. The interesting failures aren't about whose model is smarter. They're about whether the organization asking the question had already answered it for itself.
AI inherits the problem
An AI agent walks into the same warehouse carrying the same missing information the dashboards and analysts had. Ask for "revenue" and you might mean invoiced revenue, paid revenue, recognized revenue, gross revenue, or net revenue. The warehouse likely contains everything needed to calculate all five. The question itself doesn't specify which one you want, and the raw schema has no column called the_one_the_business_means.
Meaning isn't unrecoverable. Data catalogs, dbt models, documentation, and lineage can all encode it — which is exactly what a semantic layer formalizes. The gap lives in the raw tables: without that layer, the schema alone can't tell an LLM — or a new analyst — which of five reasonable definitions the business meant. "Customer," "active user," and "conversion rate" carry the same problem. Each has multiple definitions that are individually reasonable and collectively incompatible. An LLM has to resolve that ambiguity on every single question, whether or not anyone told it there was a choice to make.
This isn't hallucination
The failure mode deserves precision. "The AI hallucinated" is the wrong diagnosis, and the wrong diagnosis leads to the wrong fix.
Hallucination is a model inventing something that isn't there: a table that doesn't exist, a column it made up, a fact with no source. That's not what's happening when someone asks for "monthly active users."
"Show me monthly active users"
Interpretation A
Interpretation B
The model isn't fabricating anything in either case. It's making a plausible, defensible interpretation of a genuinely ambiguous term. That's what makes this failure mode more dangerous than a hallucination: the output looks exactly as credible as the correct one would have. In a narrow sense, it is.
Anthropic ran into this too
Anthropic's data science and engineering team measured this class of problem directly. They published an account of how they built self-service analytics with Claude.
Reported accuracy, by approach
Without skills directing Claude to check a governed semantic layer first, accuracy on Anthropic's own evals didn't exceed 21%. They then gave the model raw retrieval access to thousands of historical SQL queries from dashboards and notebooks, on the theory that a record of already-answered questions should help. It moved the number less than a percentage point. When they investigated, they found the correct precedent was present in that history for roughly 80% of the questions the model still got wrong. Having access to the right example didn't mean the model used it.
Skills that required the model to check a human-curated semantic layer before writing any SQL worked. That took accuracy above 95% in aggregate, and close to 99% in some domains. Anthropic also tried having an LLM generate the semantic layer's definitions itself. That was net-negative compared with a smaller, human-curated one.
These are Anthropic's own reported figures from their own systems. Not an independently reproduced result. They shouldn't be read as a universal accuracy number for every LLM analytics setup.
Two other figures from the same account belong next to the 21%. Anthropic reports roughly 95% of its business analytics queries now run through Claude at about 95% aggregate accuracy — so this is not a story about abandoning AI analytics. Separately, in a public interview, CFO Krishna Rao described the finance team building roughly 70 finance-specific skills rather than feeding the model raw questions. Reports that used to take hours now take around 30 minutes, with a human reviewing at the end.
Skills and semantic layers aren't the same mechanism. They come from the same instinct: the model gets reliable at the exact point where humans write down what they already know.
Why historical SQL and prompting aren't enough
Paste the definition into the prompt, or show the model queries analysts wrote before. Neither holds up, and not primarily because the model is unpredictable. A prompt is not a durable governance mechanism.
A prompt communicates a definition once, in one conversation. It has no owner, no version history, no check on whether it's still accurate after the schema changes underneath it. A production analytics system needs the definition to persist across sessions, be attributable to someone, and execute the same way every time — not restated from scratch by whoever happens to be prompting that day.
Historical queries have the same shortfall from a different angle. They're evidence, not a source of truth. They imitate style, but they also imitate every inconsistency that was already there — if two analysts defined "active" differently in the past, the model now has two contradictory precedents and no signal for which one is current. Useful context. Not something a system should treat as authoritative on its own.
The source-of-truth problem
One question sits underneath all of this: where does the organization actually define what a metric means? Not where it can be inferred from the schema — where it is written down, owned, and treated as authoritative. A source of truth can be a governed metric definition, a documented business rule, or an authoritative dashboard with a named owner. Without one, every query, dashboard, or AI agent reconstructs the definition on its own. Each reconstructs it differently.
A semantic layer is one way to implement that source of truth — a scoped, named, owned definition with executable logic that a model retrieves instead of inventing from raw tables. "Revenue" can legitimately mean gross, net, recognized, or collected depending on who's asking. A semantic layer doesn't force one answer. It makes each variant explicit and attributable instead of silent.
Several approaches exist for building this governance layer: semantic layers, metric stores, dbt-based approaches, and others. Each makes different tradeoffs. A head-to-head comparison is a separate piece. What matters is that one of them exists.
Why this wasn't urgent until now
None of this is new. In the 2000s every serious BI tool shipped a governed definition layer and treated it as the product: Business Objects had its Universe, Cognos had Framework Manager, MicroStrategy had its metadata layer. You did not query the warehouse directly. The definitions were the thing you bought.
Then cloud warehouses made that abstraction feel like overhead. Databases got fast enough that analysts could write SQL straight against them — quicker, cheaper, and far more flexible than clicking through a vendor's modelling interface. For roughly a decade, "just write SQL" was the right call. Definitions drifted between analysts, dashboards occasionally disagreed, and that was survivable, because humans are slow. When two numbers conflicted, somebody noticed in a meeting and asked.
The people who tried to sell the fix early were right about the problem and early on the timing. Benn Stancil argued for a metrics layer in 2021, then wrote about why those startups couldn't get traction. Fivetran's George Fraser made the structural version of the point: horizontal infrastructure is hard to sell on its own. Looker won because it bundled LookML with a tool people already wanted. Transform sold a standalone semantic layer and was acquired by dbt, whose MetricFlow engine came from that deal. dbt is now running the bundled version — governance as an upsell to transformation teams already running on the platform, not a separate vendor relationship.
The demand curve finally moved. The consumer changed, not the technology. A human analyst asking one question a day could absorb ambiguity. An agent asking hundreds cannot.
Why AI makes this bigger
A human analyst asks a question, writes one query, reviews the result, and notices when something looks off. An AI agent receives a question, generates a query, runs it, and iterates — re-querying, compounding steps, producing a full analysis across many questions with no one reviewing each individual step.
Same ambiguity, different scale
A human analyst
An AI agent
The real shift is that AI doesn't introduce a new way to write SQL. It changes the scale at which a semantic mistake, once made, gets repeated. Governance matters more because there's less human attention on each query the model writes.
What a governed AI analytics system looks like
Put the pieces together and the shape of a governed system looks like this. Select any node to read what it is, what it contributes, how it fails, and who owns it.
How a governed context layer gets built — select any node
Source
Warehouse & schemas
Tables, keys, row counts, and every query that has ever run.
- What it is
- The literal shape of the data: table and column names, types, primary and foreign keys, row counts, partitioning, and the query history sitting in the warehouse’s own logs.
- What it contributes
- The only fully mechanical input in the whole pipeline. Nobody has to write it down; it can be read directly.
- How it fails
- It describes structure, never intent. A column called
is_activetells you a boolean exists. It does not tell you which of three business meanings of “active” that boolean encodes, or whether it still means what it meant two quarters ago. - Who owns it
- Data platform / DBA
Postgres · Snowflake · BigQuery · Athena
Source
BI & dashboards
Explores, usage, and the queries people already trust.
- What it is
- Existing dashboards, saved explores, scheduled reports, and — most valuable — which of them people actually open on a Monday morning.
- What it contributes
- Revealed preference. A dashboard that six executives check weekly is a stronger claim about what a metric means than any document nobody reads.
- How it fails
- Dashboards disagree with each other, and nothing in the tool records which one won. This is the exact artifact that produced three different “active user” counts before any model was involved.
- Who owns it
- Analytics / BI team
Looker · Metabase · Sigma · Power BI
Source
Modelling code
Metrics, dimensions, joins, and entities already in code.
- What it is
- Transformation and modelling logic that already exists in version control: dbt models and tests, LookML, MetricFlow definitions, and the joins someone already got right.
- What it contributes
- The highest-fidelity source in the set. It is executable, diffable, reviewable, and already carries an implicit owner in the commit history.
- How it fails
- Coverage is partial and uneven. The metrics that got modelled are the ones someone had time for, not the ones the business asks about most.
- Who owns it
- Analytics engineering
dbt · LookML · MetricFlow
Source
Docs & tribal knowledge
Policies, caveats, and what the analysts simply know.
- What it is
- Written policy, close checklists, metric caveats, Slack threads where someone explained the rule once, and the knowledge that currently lives only in one analyst’s head.
- What it contributes
- The why. This is usually the only place that records the exception, the restatement, or the reason a metric changed definition after a product launch.
- How it fails
- Unstructured, unversioned, and non-executable. An agent cannot reliably act on it, and when the person leaves, it leaves. This is the layer that decays fastest and most silently.
- Who owns it
- The domain expert, not the data team
Notion · Drive · wikis · Slack
Stage 01
Collect what already exists
Read each system in the shape it natively speaks.
- What it is
- A connector per source that reads it on its own terms — introspecting a schema is not the same operation as parsing LookML or crawling a docs space.
- What it contributes
- An honest inventory before any interpretation happens. You cannot reconcile definitions you have not yet found.
- How it fails
- Skip it and you build the governance layer from whatever one team remembered to mention, which reproduces the original problem at a new altitude.
- Who owns it
- Platform / the pipeline itself
Stage 02
Propose definitions
Turn that evidence into candidate metric definitions.
- What it is
- Convert raw evidence into proposals: candidate metrics with a name, a grain, a formula, and a citation back to whichever source suggested it.
- What it contributes
- Speed on the mechanical half. Drafting a candidate from four sources is exactly the work a machine should do.
- How it fails
- This is the stage teams try to automate end-to-end, and the one Anthropic reported as net-negative. A proposal is not a decision. Ship auto-generated definitions without human review and you have industrialised the guess.
- Who owns it
- Machine drafts, human decides
Stage 03
Reconcile
Check new evidence against definitions that already exist.
- What it is
- Compare each proposal against the definitions already in force. Same metric, same meaning? Same name, different meaning? A genuinely new metric, or a fourth spelling of one you have?
- What it contributes
- The step that actually kills the “how many users” problem. Collision detection is the whole point: it forces the conflict into the open where a person has to resolve it.
- How it fails
- Without it the layer accumulates synonyms. Three definitions of “active user” in a governed store are worse than three in scattered dashboards, because now they all look official.
- Who owns it
- Metric owner, with the data team implementing
Stage 04
Validate
Prove references and logic resolve before agents use them.
- What it is
- Mechanical checks before anything is published: every referenced table and column exists, joins resolve at the stated grain, the SQL compiles, the owner field is populated, and the freshness stamp is current.
- What it contributes
- The gate that turns a document into a control. It is also where abstention is enforced: a metric that fails validation is unavailable rather than silently wrong.
- How it fails
- This is the stage whose absence produces the 95%→65% decay curve. Definitions do not break loudly. The schema moves underneath them and nothing tells you until the answers have been wrong for a month.
- Who owns it
- Automated, alarming to a human
Output
Written definitions
definitions/*.md
- What it is
- Plain-language meaning, the owner’s name, known caveats, and the analyst context an agent can search when a question does not map cleanly to a metric.
- What it contributes
- Answers “what does this actually mean, and who decided that?” — the question a number alone can never answer.
- How it fails
- Free-form text is easy to write and easy to let rot. It needs the same review cadence as the executable half, and it earns its keep only if it is retrievable.
- Who owns it
- Domain expert
Output
Executable metrics
semantic-layer/*.yml
- What it is
- Metrics, joins, grains, filters, and segments in a structured form a machine can validate and compile into SQL.
- What it contributes
- The retrievable source of truth. The model stops deciding what “monthly active users” means and starts looking it up; the number moves only when the data moves.
- How it fails
- Executable does not mean correct. A precisely wrong definition is more dangerous than a vague one, because everything downstream now agrees with it.
- Who owns it
- Data team implements, domain expert approves
Business teams own what a metric means; data teams implement it. Definitions must be executable, not parked in a wiki. Someone owns each metric and updates it as the business changes. High-stakes outputs — anything that reaches a decision, a forecast, or a board deck — get a human check before they're acted on. When no governed definition exists for a question, the system says so rather than improvising one.
A source of truth decays
There is no finish line. Anthropic tracked offline accuracy falling from roughly 95% at launch to about 65% inside a single month — before they reclassified staleness as an engineering problem rather than a documentation problem. Nothing broke in the system. The business moved and the definitions stood still.
That curve is also the cleanest argument against the obvious shortcut: letting the model write the definitions itself. Anthropic tried it — bootstrapped the semantic layer by auto-generating metric definitions from raw tables and query logs — and reported it as net-negative against a smaller, human-curated set. A model can read the schema. It cannot know that "active customer" changed meaning last quarter because the product team shipped a retention flow. The person who owns that metric knows.
What keeping it alive requires
01
A named owner
Usually the domain expert, not the data team. Growth owns “active user.” Finance owns “recognised revenue.”
02
Versioned like code
A definition changes when the business changes, and the change is reviewable by someone who would notice.
03
Loud deprecation
Retired metrics get marked, not quietly left in place for an agent to find and use anyway.
What this looks like in practice
Quarterly revenue has two versions. Book it on invoice and you get one number. Book it on cash receipt and you get another. Both queries run against the same warehouse. Both are valid SQL. Only one is the number a controller can close a quarter on — and nothing in the question specified which.
Where to start
If you're evaluating AI analytics right now, the right first move is much smaller than a platform decision.
Six moves that do most of the work
The dangerous failure mode isn't the model hallucinating. It's the model being confidently, plausibly correct about the wrong interpretation.
Back to the question
HOW MANY USERS? doesn't need better SQL. Marketing, Finance, and Growth were never disagreeing about how to write the query — they were running three different, equally legitimate definitions of "user," and the database dutifully answered whichever one it was given. It would have done the same for a person, a dashboard, or an AI agent.
AI isn't breaking analytics. It's exposing how fragile analytics already was — how much of it ran for years on tribal knowledge, one analyst's memory, or whichever dashboard happened to load first. The future of AI analytics won't be decided by who generates the most SQL. It will be decided by who preserves the meaning behind the SQL.
The 21% at the top of this page is not an argument against AI analytics. Anthropic runs roughly 95% of its business analytics queries through Claude today. The 21% is what that same system is worth before anyone writes down what the words mean.
FAQ
Why can an LLM's SQL be correct and the answer still wrong?
Because correctness has three layers: the query can parse and run, the joins and logic can be sound, and it can still answer a different business question than the one intended. An LLM that isn't told which definition of a metric to use picks one silently. See the three levels of correctness above.
Is this the same thing as an LLM hallucinating?
No. Hallucination usually means inventing something that doesn't exist, like a table or a fact. Here the model isn't inventing anything — it's making a plausible, defensible interpretation of a genuinely ambiguous business term, which is what makes the wrong answer look credible. See this isn't hallucination above.
Can better prompting solve the problem?
Not reliably on its own. A prompt can state a definition, but it doesn't guarantee the model applies it the same way every time, and it doesn't give the model an executable, reusable source of truth to check against. See why historical SQL and prompting aren't enough above.
Does a governed definition layer stay accurate on its own?
No. Anthropic reported offline accuracy drifting from about 95% at launch to about 65% within a month, because the business changed and the definitions didn't. Each metric needs a named owner, review when it changes, and clear deprecation when it's retired. See a source of truth decays above.
What does a semantic layer actually provide?
A governed, reusable definition of a business metric, its owner, and its executable logic, so the AI retrieves the organization's definition instead of inferring one from raw tables each time it's asked. See the source-of-truth problem above.