The "Chat-With-Your-Data" Trap: Why AI in BI Needs a Semantic Layer

Last week I wrote about helping a client replace Tableau with Streamlit on Snowflake, built by Claude Code. The response was better than I expected, and a good chunk of it was people poking at the same soft spot: if Claude Code is building the dashboard, how do you know the number on it is right?

Reading back through that post, I get why. There's a clause in the second paragraph that the entire thing depends on, and I gave it six words: "with a well-governed data model underneath." That's the oversimplification of the century. So people filled in the blank with the worst version, which is a business user staring at a blank prompt box with raw access to a warehouse. That is a genuinely bad idea and it's worth writing about... it just isn't what we built.

So this post is those six words. What "well-governed" actually has to mean before you point an agent at your data, and why I don't think the answer is keeping AI out of BI.

Short version: exploration should be probabilistic, publication shouldn't be, and the semantic layer is where you draw the line.

Everybody wants to chat with their data

All of my clients are interested in "chatting with their data." In practice that has meant connecting their warehouse directly to an OpenAI or Claude subscription and calling it a platform. For a business user, that seems simple enough, and frankly it should. It's like driving a car without ever popping the hood... which is a compliment to the car, not an insult to the driver.

The problem is when there's no engine under there. An agent pointed at raw tables is doing archaeology. It reads table names, column names, and whatever comments somebody wrote back in 2023, then fills in the rest with guesswork. It doesn't know that "active customer" excludes trials. It doesn't know that finance closes the month on the 3rd, so every calendar-month rollup comes up two days short. It doesn't know which of your four order tables is the one anybody actually trusts.

So it gets things wrong, which is bad but survivable, because people tend to catch an obviously wrong number. What's worse is that it gets things differently wrong for two people who asked the same question in different words. Now there are two numbers floating around in two Slack threads, no way to settle it, and stakeholders who are learning to distrust the whole thing.

The ambiguity was already there

Here's the part I think gets lost when Data Practitioners (me very much included) get defensive about this: human analysts make these exact mistakes. Fanned-out joins, wrong grain, double-counted refunds, a filter somebody added for one stakeholder back in 2024 that quietly never came out. I've shipped a few of those.

What's changed isn't accuracy so much as volume and visibility. An analyst gets the grain wrong once a quarter inside a workbook nobody diffs. An agent will do it forty times before lunch, in front of the exec team. The ambiguity was already sitting in the data model. Chat just started asking about it.

Which stings, but it's also the cheapest audit you'll ever get. Every question your agent fumbles is a definition your org never actually agreed on.

The chicken and the egg

There's a catch-22 in BI that I run into on nearly every engagement. A team spends significant time and resources gathering data from all corners of the business, and then the stakeholders aren't entirely sure what they should be asking of it. This is because people don't know what they want until they can see what's there.

Prompting is really good at that phase. Let stakeholders wander around, ask bad questions, get partial answers, and figure out which of their questions actually mattered. Most of what they turn up never needs to be published anywhere, and that's fine.

What makes it safe isn't the agent being clever. It's the boundary: nothing gets trusted org-wide until it's been validated and defined in the semantic layer, and everything on the far side of that line runs the same fixed logic every single time. An agent exploring is allowed to be wrong. A published dashboard isn't allowed to be surprising.

There are three versions of this I keep walking into:

Pre-AI. Every question is a ticket. The analyst is the gate between a question and a trustworthy answer, not because analysts are slow, but because they're the only ones who know what "revenue" actually means in this warehouse.

AI bolted on. The stakeholder finally gets a fast loop, and unverifiable answers. Questions do get sharpened before they ever reach an analyst, which is a real win. But the analyst still owns dashboards, still owns governance, and now also owns cleaning up the numbers already circulating in Slack.

AI on a semantic layer. Same chat experience, different target. The agent queries validated metrics instead of raw tables, so "active customer" means one thing whether the answer came from an agent or a dashboard. Analysts are still the gate for anything that gets published, but a lot less reaches them, and what does tends to be a real gap rather than a chart tweak.

What I meant by "well-governed"

Roughly in order of how much it hurts when it's missing:

1. Access and redaction. Sensitive fields hidden, masked, or excluded for whoever the agent is acting on behalf of. The agent inherits the permissions of the person asking, so if your row-level security is more aspirational than real, chat is how you'll find out.
2. Tested data. Assertions on uniqueness, referential integrity, nulls, accepted values (dbt tests, Great Expectations, whatever you like). Running on a schedule and failing loudly. A wrong number should get caught by a pipeline, not by your CFO.
3. Documentation that describes the business, not the column. order_ts isn't documentation. "Timestamp the order was captured, in UTC. Not the ship date, not the recognized-revenue date" is documentation.
4. A semantic layer where the ambiguous stuff is defined exactly once. Revenue vs. sales. Fiscal vs. calendar. Active vs. registered.

The first three are table stakes, and most competent data teams have some version of them in place already. The fourth is the one almost nobody has, and it's what I want to spend the rest of this post on.

In Snowflake, that's a semantic view

CREATE OR REPLACE SEMANTIC VIEW analytics.sem.revenue
  TABLES (
    orders AS analytics.core.fct_orders
      PRIMARY KEY (order_id)
      WITH SYNONYMS = ('sales', 'transactions')
      COMMENT = 'One row per captured order. Excludes trial orders.',
    customers AS analytics.core.dim_customers
      PRIMARY KEY (customer_id)
  )
  RELATIONSHIPS (
    orders_to_customers AS orders (customer_id) REFERENCES customers (customer_id)
  )
  FACTS (
    orders.net_amount AS gross_amount - discount_amount - refund_amount
      COMMENT = 'Net of discounts and refunds.'
  )
  DIMENSIONS (
    orders.fiscal_month AS fiscal_period_label
      WITH SYNONYMS = ('month', 'period')
      COMMENT = 'Fiscal month per the finance calendar, which closes on the 3rd.',
    customers.segment AS segment
  )
  METRICS (
    orders.revenue AS SUM(orders.net_amount)
      WITH SYNONYMS = ('net revenue', 'sales')
      COMMENT = 'Net revenue. This is the number in the board deck.',
    orders.active_customers AS COUNT(DISTINCT orders.customer_id)
      COMMENT = 'Customers with at least one non-trial order in the period.'
  )
  COMMENT = 'Governed revenue metrics. Every agent and dashboard reads from here.';

And then anything that needs a number asks for it like this:

-- Agents and dashboards both query the semantic layer through this,
-- which is why the two of them can't disagree about what "revenue" means.
SELECT * FROM SEMANTIC_VIEW(
  analytics.sem.revenue
  DIMENSIONS
    orders.fiscal_month AS fiscal_month,
    customers.segment AS segment
  METRICS
    orders.revenue AS revenue,
    orders.active_customers AS active_customers
)
ORDER BY fiscal_month, segment;

Notice what isn't in that query: no joins, no SUM(), no CASE WHEN to drop trials, no date math to line up with the fiscal calendar. All of that got decided once, in the view definition. The caller just names the metric it wants and the grain it wants it at, which is about as small a decision as you can hand an agent.

The syntax will look different depending on where you build it (dbt has a semantic layer, Cube exists, Looker has been doing this with LookML for a decade), but the idea holds either way. Revenue gets one definition, it lives next to the data instead of inside a BI tool's proprietary modeling layer, and every consumer inherits it: Cortex Analyst, a Streamlit app, an agent, a notebook, a Tableau extract if you've still got one. Nobody re-derives the metric per surface.

That's how you get the chatbot's answer and the dashboard's answer to agree. Shared math, not better prompts.

Let AI write the code, not the metrics

The biggest misconception in all of this is that the LLM is supposed to be the thing calculating the number. It isn't. Its job is code generation.

An agent querying a semantic view isn't deciding what revenue means, it's picking an already-defined metric and slicing it, which is a much smaller decision and a much easier one to review. And when it drafts a Streamlit app against that same view, what comes out the other end is a versioned artifact that goes through a pull request and runs the same logic on Tuesday that it ran on Monday. (Which is exactly what we built in the last post... this one just belongs underneath it.)

Messy exploration on one side, boring version-controlled reporting on the other, the same definitions holding up both.

What it'll cost you

I'd rather say this part myself than have it turn up in the comments.

A semantic layer is real modeling work with nothing to show for it on day one. You're asking an analyst to spend a couple of weeks building something no stakeholder will ever look at directly, which is a hard sell in most orgs.

It also moves the bottleneck rather than removing it. Every question the agent can't answer turns into a request to extend the semantic layer, and that request lands on the same analyst who used to be the dashboard bottleneck. If nobody staffs that queue, the layer goes stale and everyone quietly drifts back to querying raw tables.

And it needs the same treatment as the rest of your code: version control, review, tests on the metric definitions themselves. A semantic view with a bad definition in it is worse than not having one at all, because now everything is consistently wrong and nobody's checking.

Why I still think it's worth it

Fix a dashboard and you've fixed a dashboard. Fix a definition in the semantic layer and you've fixed every question that will ever touch that metric, for every human and every agent, from here on out.

Same analyst, same hours, work that accumulates instead of repeating. That's a trade I'd make every time.

If you want the other half of this, the Streamlit setup that sits on top of it is all open source at github.com/chrishronek/streamlit-dashboards. Happy (governed) chatting! 🎈