Don’t Outsource Your Agentic Analytics Stack, Compose It

TL;DR

• “Out-of-the-box” agentic analytics systems sell some version of “connect all of your data sources here, and we’ll do the rest,” and they lock your semantic layer, business context, evaluation suite, and feedback loop inside one product. Those four things are the record of how your company understands itself, and they belong to you. Do not outsource them. A vendor selling one component you can compose with the rest of your stack is a different matter.
• Compose the rest. Rent the warehouse, the model, the agent runtime, and the chat window from whichever vendors fit your architecture; all of them get cheaper every year. Any rented component should be replaceable without rewriting the four. That is possible when the four are stored in formats your team can read, export, write to, and keep confidential. Keep them in your own repository, and both the platform’s agent and any outside agent connecting over MCP can read the same definitions.
• The always-available analyst is a goal worth having, and the out-of-the-box vendors sell you the fast track to it knowing that your reliance on them only increases over time. The four components highlighted above are essential for agentic analytics to work, and they will evolve into critical infrastructure if you intend to utilize agentic analytics over the long term. Don’t cut corners; select a composable solution, or at least one that gives you the flexibility to move toward composable in the future.

The corner-cutters are back at it again. This time, rather than connecting agents directly to their systems of record (ERP, CRM, SaaS applications, etc), they want to buy an out-of-the-box agentic analytics system that manages their semantic layer and business context and feedback loop for them. The eventual goal, which I support, is to put an automated analyst in the pocket of every decision-maker across the organization.

The original catch, which I describe in “Skip the Data Stack, Get the Wrong Answer Faster”, is that connecting AI directly to your data is insufficient to accomplish this goal. The emerging catch is the topic of this blog post: owning those four components is the only way to ensure your organization is learning as it grows rather than accumulating technical debt and vendor lock-in.

My viewpoint is simple: you should not outsource ownership of your semantic layer, context layer, evaluation suite, or feedback loop (all four defined below). You should not fall prey to the sales pitches of “out-of-the-box” agentic analytics systems which tell you the short term cost of managing your own is not worth the long-term benefit.

The hyperscalers and data warehouse companies already know the goal of an agentic analyst in your pocket is both a worthy one and one that has captured the attention of budget-holders across enterprises. That’s why they are bending over backwards to offer themselves as the one-stop solution for agentic analytics. Databricks announced Genie One in June 2026 as “an all-new agentic coworker” for every business team, on top of Unity Catalog metric views and its Genie Ontology. Snowflake sells Cortex Analyst over native semantic views, Cortex Agents, and Snowflake CoWork (previously Snowflake Intelligence), and it co-founded the Open Semantic Interchange standard for moving semantic definitions between platforms. Google took Conversational Analytics for BigQuery to general availability in June 2026, grounded in LookML or in context you author inside BigQuery. Microsoft’s Fabric Data Agents plug into Copilot Studio and Microsoft 365 Copilot. Amazon has Q in QuickSight, built on Bedrock. To be clear, I think it’s great that each of these companies has offerings up and down the analytics stack, as long as those offerings meet our criteria for being (re-)composable (see Choose the (Re-)Composable Data Stack)

The four data stack components this post discusses

Four components help to turn a language model into an analyst you can trust. This post discusses them a bunch. Skip these if you feel like you already know.

• Semantic layer (SL): The business-logic description that sits between raw database storage and the tools or users consuming data, translating complex technical schemas into familiar business terms and governed metrics.

• Context layer (CL): Everything the agent needs to know that does not reside in the schema or SL, including the domain-specific documents explaining how each department views the world, internal process documentation that explains how the data is generated, verified question-and-query pairs, precedence rules for conflicting sources, and any other information that helps the agent to understand its subject matter and audience.

• Evaluation suite (ES): A maintained set of questions with known-correct answers and known-correct evidence paths, run against the system every time the SL, the CL, the model, or the prompt changes. This is how a data team monitors whether a change to the SL/CL makes the system better or worse.

• Feedback loop (FL): The process that takes what happens in production (the questions people asked, the answers they got, and the corrections made by the people who knew better) and routes it back into the SL, the CL, and the ES. Without it the other three decay. Anthropic saw accuracy fall from roughly 95% to roughly 65% in a single month when its skill documents went unmaintained.

Figure 1. Where the four owned components sit in your data platform.

First, let me define what I mean by “out-of-the-box” agentic analytics system, as compared to a “composed” analytics system. Out-of-the-box vendors are those who sell some version of “connect all of your data sources here, and we’ll do the rest”. They bring your data into a data warehouse (or connect to the one you already have) and create a SL/CL for you, housing all of your business logic and metric definitions and source of truth designations in one place. They provide you with an agent you can use to ask questions of your data. They provide you with a FL for filtering feedback from these conversations into the SL/CL to improve it over time. Some might even offer an ES as well. A composed analytics system, on the other hand, is one which combines off-the-shelf components from cloud, data warehouse, AI, and business intelligence vendors, along with open source / internally developed tools, into a system that you operate. This is the type of system we usually recommend.

Table 1. Comparison of out-of-the-box and composed agentic analytics systems.

We believe strongly that you should be composing your agentic analytics system rather than relying on out-of-the-box vendors. Allow me to enumerate the reasons why:

The SL and CL are what make the agent work

1. Your SL/CL is the thing that makes agentic analytics feasible with a reasonable degree of accuracy. Without well-articulated business context and metric definitions, agents will flounder and provide the wrong answer as often as not. A well-crafted SL/CL enables agents to provide the right answer the vast majority of the time, explain its reasoning in a way a business user can understand, and even identify questions that the agent is not well-positioned to answer. Every benchmark I’ve seen demonstrates improvement on complex datasets when a semantic layer is incorporated. Show me one benchmark that doesn’t support this.

   Here’s two, for example, that do support it:

   • dbt Labs’ April 2026 benchmark, where raw text-to-SQL (the model writing queries against the schema directly, with no semantic layer) improved from 32.7% to 64.5% across three years of model releases while questions covered by the SL returned correct results 98% to 100% of the time depending on the model. Questions not covered by the SL provided an error message rather than a dumb guess.
   • MotherDuck’s DABStep run, where agents worked a public benchmark of multi-step data-analysis questions with and without a context layer (MotherDuck Guides, markdown documents the agent reads before it queries). Adding the context layer took agent query errors from 302 to 1 and cut the cost per run by 55%. With it, a sub-frontier model (Gemini 3 Flash) answered 418 of 419 held-out questions correctly.

2. Your SL/CL is the thing that makes the answer reproducible over time and across interfaces. You don’t want to get different answers depending on who asks, how they ask, where they ask, and when they ask. Agents, by nature of being probabilistic systems, will not arrive at the same conclusions using the same approaches consistently, unless they are provided sufficiently clear context, delivered at a sufficient level of consistency to arrive at those answers.

   You also want to prepare your company for a world in which multiple agents and models will be interacting with your SL/CL layers. Agentic analytics is only one function that AI could utilize your SL/CL documentation to accomplish. Board decks, systems integration, customer support, sales enablement, marketing automation… the applications are effectively endless. If all are referencing the same information, then the improvements that drive better marketing automation also drive better analytics, and vice versa. If the SL/CL is fragmented across systems, or worse, each function has its own system-specific agent that only has access to a different lens on the same information, then the silos and data definition conflicts that created pain in the business intelligence era will cause even worse disruption in the agentic analytics era. You want the SL/CL to be a single surface for all use cases, across functions and business lines. This unifies the organization in a way that may never be possible without an agentic analytics stack.

What the system learns should belong to you

3. You want to teach each other how to run your business. You don’t want to teach a third party. It is hard to overstate how close the context surrounding your company’s data is to the blueprint for understanding and running your business. The knowledge that accumulates in your business over time about your customers, your products/services/processes, and your shared language, is the essential thing that separates you from your competitors. Therefore, you cannot afford to allow someone else to construct and maintain your SL/CL/ES/FL. You want to be intimately involved in it yourself.

   Satya Nadella, Microsoft’s chairman and CEO, described this necessity in July 2026 in an essay he titled “The Reverse Information Paradox”. According to Nadella, “The better you want the model to perform, the more of [your organization’s] knowledge you have to feed it!…[You pay for intelligence] once with money, and again with something even more valuable: the proprietary knowledge“ you have to reveal to make the intelligence useful. Ian Macomber, Ramp’s head of analytics engineering and data science, arrives at the same place from the operator’s side: “Either context compounds within your four walls, or within someone else’s.”

4. Your business changes over time, and only your people know what changes are material. It is a seductive fiction that AI can detect material changes in your business better than you can. Only you know whether a decline in revenue is an artifact of systems migration, changing revenue line definitions, or a true shift in the business landscape that needs to be diagnosed and acted upon. Outsourcing this distinction to AI leaves open the likelihood, if not the certainty, that noise in your business data will become noise in your business strategy or operations.

   Two of the best-resourced teams in the world tested my assertion. Anthropic tried having a model generate its semantic-layer metric definitions and found that it “produced plausible-looking definitions that encoded the very ambiguities we were trying to eliminate, and was net-negative on our evals.” Ramp first bought the interpretation of its sales calls from a vendor that wrote free-text fields back into Salesforce, and then, in Macomber’s account, “the vendor’s tagging distribution shifted with no warning, in a way they couldn’t explain, debug, or revert.” Ramp could no longer reproduce its win/loss reporting. His summary: “Best case, you don’t own it. Worse case, they break it, and you can’t fix it.”

5. You do not want any of these company IP assets to be locked into a single vendor. You do not want any vendor to be able to train on these assets. You want these assets to be proprietary and portable. Clearly, the intimate information about how you measure success, what your systems of record contain, how your business operates, and what questions your stakeholders are asking is private and confidential. It is also foundational to the long-term success of an organization in an era where AI capabilities are expected to expand over time. If you believe that over the course of your lifetime, you will be able to onboard AI employees that help the organization to evolve and improve, then you must also believe that the SL/CL/ES/FL you develop now, validated against real-world utilization by your decision-makers, is in many ways the most valuable asset your company can create. You cannot allow a vendor to trap that information, and you cannot allow third parties to build systems that learn from that information, now or in the future. The most “AI-pilled” among us should be the most in favor of having humans do this work.

   Macomber lists what the full-stack vendors ask of a data team:

   • Write your entire semantic layer and business context in their tool’s proprietary language.
   • Keep all of your models, dashboards, and analyses in their tool (and their tool only).
   • Send all of your business stakeholders to their tool (and their tool only).
   • Let them leverage user activity to improve their understanding of your data and company.

   To me, this is far too much to ask.

You need to see inside the system

6. You don’t just want to know the right answer. You also want to know the source of the answer and the source of the mistake. Your role as the human in these agentic analytics systems is to utilize the provenance of the information (i.e., which data sources are referenced, the definition/method utilized for the calculation, even who wrote the definition) to determine how much to trust the result, or to identify the right way to guide the agent to a better answer for you. With a SL/CL, this information can be provided using language that is accessible to both the agent and the business user. Without it, the agent provides the business user with a loose combination of SQL syntax and data-domain-specific language to explain provenance, which is confusing and frustrating. We all benefit from these systems being transparent with regard to provenance; it allows anyone to contribute to the FL on how to answer a given question correctly in each context under which it might be asked.

   It is not enough to know that something was wrong and now it is fixed. Organizational learning requires understanding the difference between what you knew before and what you know now. It requires updating your mental model of the business so that you can easily detect problems in the future. Especially if you are interacting with AI regularly, you need to refine your AI bullshit detector so that it doesn’t slide something by you. I regularly read AI-written documentation and get overwhelmingly frustrated by the number of times it states something obvious as if it were genuinely groundbreaking, or on the other hand discusses something compressed and jargon-filled as if it has already been effectively contextualized. AI has no theory of mind, nor sense of the audience, things that humans develop from the time they are born. The SL/CL documentation grounds the agent in how the organization sees itself currently. As the SL/CL evolve, the FL allows the organization to update its collective viewpoint given new information. A third party should not be the cultivator or editor of your organizational viewpoint.

   Every internal agentic analytics system blog post I can find makes provenance a first-class output. Meta’s analytics agent displays the SQL behind every result. Anthropic ends every response with a footer describing whether the answer came from the semantic layer, a governed table, or raw exploration. Ramp normalizes each run into standardized steps (i.e, the agent reads the domain document, views the dashboard, reads the semantic view, executes SQL, synthesizes the answer) so that a data team reviewer can see the path and grade it.

   For the AI Labs, they seem to have settled on the “changelog” as the correct structure for the FL. At Anthropic, a scheduled agent “scans stakeholder channels every few hours for similar correction language, drafts a one-line fix to the relevant reference doc, and opens a PR tagged to the domain owner,” and roughly 90% of the team’s data-model pull requests now include a change to the corresponding skill document in the same diff. The correction, the definition it changed, and the person who approved it are part of the changelog. OpenAI’s internal agent saves corrections as memory records so that the next question benefits from them. In a vendor’s tool, the same corrections might occur, but you likely have no visibility into them, control over them, or ability to take them with you when you leave.

7. Only you can determine what matters to the humans in your business. You need visibility into what specifically your system is asked, what it gets right, and what it gets wrong. To an agent with no guidance about what matters to you, everything matters equally, so nothing stands out. Your ES and FL are where you record what does matter: the questions your stakeholders ask, the answers that were correct, and the places the agent got it wrong. Owning that record gives you three things a vendor’s console will not: (1) You see which questions people ask, and therefore which parts of the SL/CL are thin. (2) You see which answers were wrong and can trace each one back to the errors or omissions that caused them. (3) And after every change, you can rerun the same validated questions to check whether the system improved or regressed.

   DoorDash’s evaluation set is a textbook example of an in-house ES. More than 1,900 evaluations across 19 domains “have been contributed by both domain owners and everyday users; a domain owner vets every user contribution before it counts.” New documentation and join patterns “are marked inactive and are subject to passing an evaluation threshold before a domain owner promotes them,” and when Vera (DoorDash’s in-house agent) detects a change in an upstream table or document it depends on, that change waits in the same approval queue until a domain owner accepts it. Anthropic operates in a similar way: a domain owner “can’t announce the agent to their stakeholders until their slice of the eval set clears some threshold (we initially used ~90%).” Macomber’s rule for what to do when a test fails is: “Assume the models are never wrong, your context is just underspecified.” The ES is critical to understanding those gaps and filtering feedback into the SL/CL.

Choose the (Re-)Composable Data Stack

None of this is an argument against buying software. Every company in Table 2 (see Appendix) rents its AI model, most rent the data warehouse, and several rent the agent runtime and the chat surface too. The composed stacks we typically recommend include a variety of vendor-sourced components: ingestion, storage, the catalog, the model gateway, the trace store, and the delivery surfaces, from whichever vendor fits your existing architecture.

Figure 2. The Agentic Analytics Stack: What to Rent and What to Own.

By (re-)composable, I mean a stack whose rented components can be swapped without rewriting the SL, CL, ES, or FL. These four criteria should be used to determine whether a given architecture for your SL, CL, ES, and FL qualifies as (re-)composable and does not therefore raise red flags:

1. Readability: Can a person on your team open the SL and CL as files and read them without the vendor’s product running? Can a third-party agent read the same context the vendor’s interface reads, over MCP, and get the same definitions? Markdown, SQL, and dbt YAML pass; a proprietary model viewable only in a browser does not.

2. Portability: Are you able to export the entire semantic model directly to a version controlled repository? If that interface does not exist, you are facing the prospect of lock-in.

3. Writeability: Are you able to write, or ask AI to write, aspects of your SL/CL/ES, with the promise that changes immediately propagate to your data warehouse and your agentic analytics system?

4. Confidentiality: Does every contract you sign state that the vendor does not use your definitions, corrections, questions, or traces (the logged record of which documents, tables, and queries the agent used to reach each answer) to improve its product for other customers?

A platform-native agent over a context layer you keep in portable formats passes all four, which is why most of the companies we talk to end up on an existing data platform rather than building the components themselves. Snowflake’s semantic views, Databricks metric views, MotherDuck’s schema comments, and BigQuery’s authored context are all reasonable places to publish definitions you maintain in your repository, provided they pass and continue to pass the four criteria above. You then get to choose whether to utilize the platform’s agents or external agents that connect over MCP.¹

Final thoughts

The always-available analyst is a goal worth having, and we help clients compose agentic analytics systems across a range of data platforms and architectures. The model, the warehouse, the connectors, and the chat window get cheaper every year; the semantic layer, context, evaluation suite, and feedback loop become critical infrastructure. Out-of-the-box vendors sell the fast track and will tell you the upfront investment is not worth it, since keeping the four components current is ongoing work they can absorb at reasonable cost. My viewpoint is that the effort you put into these four components is the same effort you put into any proprietary process or IP worth refining, and outsourcing it costs you strategic flexibility and the ability to improve over time. You may save money in the short term. In the long term, I believe you will regret cutting corners.

Our AI strategy practice runs assessments to help compose data systems that deliver agentic analytics that maximize ROI and minimize risk, including the risk of vendor lock-in. Please reach out for a consultation on your data architecture options as you consider composing your agentic analytics system.

Appendix: Don’t just take it from me

We reviewed fourteen first-party engineering accounts of agentic analytics systems companies run internally, and the accounts provide evidence for what I argue above. Every one of them rents the model. Most rent the warehouse. Every one of them built the SL or a functional equivalent, the CL, the ES, and the FL. Table 2 lists nine of them.

Table 2. Nine internal agentic analytics systems, and which components each company rented and built.

Two important themes to take away:

1. Every team tried shortcuts first, but every shortcut failed. Most started with plain text-to-SQL over the schema and abandoned it when accuracy collapsed on real questions. Anthropic then tried retrieval over its corpus of past queries and reported that it “moved accuracy by less than a point.” Anthropic also tried letting the model write its metric definitions, with the net-negative result reported above. Ramp tried buying the interpretation of its sales calls and could not repair it when the vendor changed it. The architecture in the table is what remained.

2. The build vs. buy choices spanned team size. Anthropic’s writeup has five authors. LangChain’s data team was one person at the start and is three now. incident.io’s writeup has one author, its lead data analyst. Each company built documents, definitions, SQL, and a question set on top of a rented agent runtime and a rented model. Nothing in the writeups is outside the scope a mid-market data team (with or without external support) can achieve.

Sources

• Ian Macomber, The Shape and Feel of the Post-AI Data Stack, August 30, 2026.

• Jason Ganz and Benoit Perigaud, Semantic Layer vs. Text-to-SQL: 2026 Benchmark Update, dbt Developer Blog, April 7, 2026.

• Ulmer, Döhmen, Matson, and O’Brien, Context belongs in the warehouse, MotherDuck, July 2026.

• Satya Nadella, The Reverse Information Paradox, July 2026.

• Ian Baldwin, Jacopo Himberg, and Akshat Khandelwal, Inside Vera, DoorDash’s Data Agent, September 16, 2026.

• Bonnie Xu, Aravind Suresh, and Emma Tang, Inside OpenAI’s in-house data agent, January 29, 2026.

• Anthropic, How Anthropic enables self-service data analytics with Claude, June 3, 2026.

• Analytics at Meta, Inside Meta’s Home Grown AI Analytics Agent, March 30, 2026.

• Matteo Vasirani and Cynthia Joseph, How we built an internal data analytics agent, GitHub Blog, June 19, 2026.

• Uber Engineering, Unlocking Financial Insights with Finch, July 17, 2025.

• Faiz Hilaly, Cesar Duran, and Jay Sobel, Meet Ramp Research: Our Agentic Data Analyst, September 18, 2025.

• Emily Hawkins, How LangChain Built an Agent-First Data Stack, July 27, 2026.

• Navo Das, Institutional knowledge doesn’t scale: Building an agentic data analyst, incident.io, August 3, 2026.

• Snowflake, Open Semantic Interchange initiative press release, September 23, 2025.

• Snowflake, Semantic views overview, Snowflake documentation; Snowflake, Manage Snowflake Semantic Views with a New dbt Package, October 2025.

• Databricks, Metric views, Genie Ontology, and Genie Space API, Databricks documentation.

• Google Cloud, Build a data agent using HTTP and Python, Conversational Analytics API documentation; Google Cloud, Version control and deploying changes, Looker documentation.

• Microsoft, Fabric data agent API is now public, Microsoft Fabric blog.

• Databricks, Genie One press release, June 2026; Google Cloud, Conversational Analytics in BigQuery now GA, June 30, 2026.

Further reading

• Benn Stancil, The context layer, August 29, 2025.

• Jason Cui and Jennifer Li, Your Data Agents Need Context, Andreessen Horowitz, March 10, 2026.

• Shapor Naghibzadeh, The post-AI data stack is right. Most companies shouldn’t have to build it., QueryStory, September 3, 2026. The dissent: Naghibzadeh, QueryStory’s co-founder and CEO, argues that most companies should buy this stack, and his company sells it.

• Apache Software Foundation, Apache Ossie (Incubating): The New Name for Open Semantic Interchange, July 10, 2026.

Related CorrDyn posts: Skip the Data Stack, Get the Wrong Answer Faster; The Context Layer: What to Build, What to Skip, and Where to Start; MCP in the Enterprise: Cost, Security, and Workload Routing; The Maintenance Trap: Why AI-Accelerated Data Teams Feel Slower.