Listen on
Overview
Agents are becoming the main consumer of the data platform, and the data team’s job has changed less than that suggests. It still curates what the organization knows and authors the context that makes that knowledge usable. The reader of that context has changed. Business intelligence tools exposed semantic-layer metadata to people through an interface, while an agent pulls schema information, semantic-layer notes, and documentation through a back end, on the fly. That shift changes what a data team builds, who it hires, and how it learns what the business is asking.
In this episode of Eventual Consistency, Ross Katz (Principal and Data Science Lead at CorrDyn) and Jason Bradwell (founder of B2B Better) work from Katie Bauer’s split between platform work, which prepares the supply of data, and distribution work, which gets it to the people who decide. Ross adds a second axis for who consumes it. A data team that serves agents needs a platform for agents and a distribution function for agents, alongside the human versions it already runs, and the agent half is work most teams have not started.
The warning signs arrive before most teams expect them: two agents returning two numbers for the same question, a reported metric nobody will sign, a compute bill that outruns what the same questions cost per BI seat. Whether to buy, build, or bring someone in depends on how stable and how specific the work is. Production has become cheap, so judgment about what an agent produced is now the scarce skill. Tacit knowledge splits into three kinds, and an analyst can write down only the first. In the What We’re Watching segment, Ross reads the Otis et al. study of Kenyan entrepreneurs as evidence that AI helps novices on well-defined tasks and experts on open-ended ones.
Key Takeaways
The data team’s job is unchanged, and its consumer is new
Ross has always described data work as knowledge curation and insight excavation: sensing events no person witnessed, such as a click, a checkout, or a visit after an ad, and turning them into something the organization can act on. The metadata that work depends on is now called context. Semantic layers were once authored so that business users could explore data in a BI interface. Agents read the same context through documentation, APIs, schema information, and notes in the semantic layer, so it has to be written for a different reader. dbt and Hex argue data teams are well placed to do this. Ross’s caveat is that the surface where agents get their context is contested, and other functions inside the company want part or all of it.
An agent-ready org chart has four quadrants
Katie Bauer divides data work into platform and distribution. Ross adds the consumer, which gives four roles. Platform for humans is the warehouse, core data modeling, and BI a company runs today, a shrinking share of the work that still needs people. Distribution to humans is the relatively senior, business-facing people who guide executives through hard, ambiguous questions and whom an executive can Slack when an AI answer feels wrong. Platform for agents is the net new work: deciding how context reaches the agent, whether through an MCP server embedded in the warehouse or another route, whether access control runs through the user or a service account, and whether monitoring and evaluation are in place. At a certain scale Ross sees that as one or more full-time jobs.
Distribution for agents replaces the demand signal the data team lost
Data teams used to learn what the business needed from tickets and direct questions. Now people ask an agent instead, and that signal has mostly disappeared. Ross wants someone whose job is to go and find it again, by interviewing people who use AI, watching over their shoulder as answers come back, and flagging wrong ones to the platform team. That role also reconciles the multiple sources of truth that appear when agents answer questions nobody asked before, and feeds them into the data model’s enhancement queue. It might be the same person who handles distribution to humans. Without it, the team is maintaining a platform and setting its roadmap with no view of how it is used.
An evaluation set stops being a test once you tune against it
The smart first move is to hand agents the semantic information already used for business intelligence and evaluate their answers against a gold set of expected questions and responses. A team already doing that is in a good place. The catch is that the questions people can ask always outnumber the evaluation set. Once the semantic layer is tuned to pass that set, the team has overfitted to it, the same way an AI lab can overfit to a benchmark that no longer reflects real work. Ross’s remedy is to keep expanding the suite and to hold out a test set the team never tunes against.
Five failures will push companies to invest in data foundations
Ross expects the urgency to come from specific incidents. Two people ask their agents for the same number, get two answers, and neither can explain how the agent reached it. An executive acts on a confidently wrong answer, which is easy because reinforcement learning from human feedback has made these models very good at sounding right. The compute bill climbs past what the same questions used to cost per BI seat. A regulated, audited, or externally reported metric depends on an AI-produced number and nobody will sign for it. The person who held the business logic in their head leaves. How soon any of this hurts depends on the cost of an error: a solopreneur can run on confident wrong answers, and an operating company with many people accountable for its results cannot.
Buy, build, or bring someone in depending on how the work will change
Ross’s test for an agentic AI platform is how stable and how specific the work is. If the need is generic and will not change, buy the SaaS product, amortize the cost, and keep the option to build later. If it is stable but specific to your business and will exist for the long term, build an internal team that holds that domain expertise. If the work is spiky, a one-time custom build, or the requirements are not yet known, an external consultant can make sense. Ross gives three reasons companies bring one in at all: nobody inside has done the work before, the team lacks the capacity alongside run-the-business analytics, or the team lacks the internal credibility to make the case to executives.
Judgment is the bottleneck now that production is cheap
AI makes producing analytical assets easy, and that moves the bottleneck to judgment: consuming what an agent presents, asking the right questions, and noticing what fails the smell test or is missing context. The same tools produce B- or C-level work in almost no time, and some organizations are content with passing grades. Ross’s view is that AI lets him do A-plus work with roughly 75 to 80 percent of the effort, so his standard for done has gone up. An AI-looking slide deck can be good enough. AI-generated documentation that nobody with domain and technical expertise reviewed, then used a hundred times a day by executives, is a landmine. Early-career data people swing between perfectionism that cannot keep pace and cutting the wrong corner, which is why the data leader has to set the standard for done.
Tacit knowledge comes in three kinds, and each belongs to a different level
Anyone who already wrote dbt YAML documenting business logic can do context engineering, Ross argues. The first kind of tacit knowledge is the unwritten fact, such as an acquisition that distorts one year’s revenue or a fiscal-calendar quirk, and analysts should build it into the semantic model. The second is pattern recognition, knowing a number looks wrong, which he expects of data leaders and which he places outside what AI can do for the foreseeable future. The third is judgment about which questions matter and when to override an answer, which the best data leaders and executives supply. The unsolved problem is where written knowledge goes when it belongs to a customer or the whole business rather than a column or table. MotherDuck already lets dbt’s persist_docs push column- and table-level documentation into the warehouse, where the MotherDuck MCP server can query it. MotherDuck Guides store organization-level documentation in the warehouse too, and Ross expects Databricks, Snowflake, and BigQuery to follow.
AI helps novices on defined tasks and experts on open-ended ones
In the What We’re Watching segment, Ross takes on the question of whom AI helps. On well-defined tasks, novices gain the most, while experts gain speed at a small cost in quality they may not care about. On open-ended tasks, he reads the Otis et al. study of Kenyan business owners as showing that inexperienced people lose ground. His explanation is that open-ended work is an iterative game: with weak judgment, each round of AI output compounds a wrong turn you cannot detect. Experts on open-ended problems see the largest absolute gains. His advice for anyone investing in their skills is to get better at ambiguity through creative, open-ended projects that force you to use judgment and make mistakes.
Related: Don’t Outsource Your Agentic Analytics Stack, Compose It | The Context Layer: What to Build, What to Skip, and Where to Start | Managed Data Team | Episode 27: What the Prefect-Dagster Merger Means for Your Orchestration Stack
Full Transcript
Jason: Welcome back to another episode of Eventual Consistency and today I’m joined by Ross Katz, Principal Data Science Lead at CorrDyn. In this episode we’re going to be talking about what happens to the data team when agents, not people, become the main consumer of your data platform. Ross makes the case that the job itself hasn’t changed all that much. It’s still curating knowledge and authoring context, but the consumption point has moved, and that changes what you build and who you hire. We cover the org chart that Ross would draw today, the warning signs that tell you it’s time to invest, and when to build this in-house versus bring someone in, and why judgment is now the bottleneck rather than production. Here we go.
Jason Bradwell: So Ross, let’s start with the big one. How has the role of the data team and the data practitioner evolved in your opinion since the advent of AI?
Ross Katz: Yeah, let me just talk a little bit about what I’ve always viewed the role of the data team and the AI team to be. This is something that I’m asked to talk about a lot because a lot of people ask me what I do. The idea is that, as a human being, all of the knowledge exists in our head. Sometimes we can remember it, sometimes we can’t remember it. But when you start to work in an organization with other people, there’s knowledge that exists in multiple people’s heads. And when you start to exist in an organization with multiple people where a lot of the business that’s happening is not happening directly between one human being and another human being, so nobody is present for the actual event or set of events that you care about, just as an example, somebody visiting your website, somebody clicking on a button on your website, somebody going to check out, somebody viewing an advertisement about your company and choosing to visit your website after viewing that advertisement. All of these things happen for your business, in your business, but there’s not a person present to sense it. And so what you need are systems and processes for sensing what’s happening in the external world and then responding as a business to whatever the situation or the context is. There’s the immediate response, which broadly speaking is the realm of traditional software. The people who own the website: when you click this button, this is what happens. And then there’s the long-term response of the organization. How do you curate the knowledge that the organization is gathering over a long time horizon and excavate insights from that knowledge in order to meet the needs of all of the different people across the organization who need to make decisions or take actions that respond to the ever-evolving environment in which your business operates? So the role of the data team has always been that knowledge curation, that insight excavation. Now, what’s different now, and this is Katie Bauer’s blog post, who was analytics at Twitter and now is the head of analytics at Hex. She talks about two types of roles for the data team. There’s the platform role, which is preparing the supply of data, making sure that all of the information is available to be delivered to people who need to make decisions and take actions. And then there’s the distribution role of getting the information to the people who need to make decisions and take actions. I would say that what AI is doing to the role of the data team is it’s altering the nature of the platforms that need to get created in order to serve a different set of consumers for the data on the distribution side. The platforms that data teams have built previously have been directed at human consumers. People are going to view this dashboard. People are going to consume the outputs of this recommendation engine. People are going to make this decision at this time, which means that this piece of information needs to be present in this system at this time. But what’s happening now is that agents are now the primary or increasingly primary consumers of the supply of information that is present in the platform. So all of the elements that needed to be considered when you were serving information to humans now need to go into the platform as it pertains to agents. And as we discussed with Dave Yaffe in our previous episode, a lot of the things that you need to worry about there are governance concerns. But the one thing that has stayed consistent over time, and we’ve used different words to describe it, is that there needs to be both the presence of the data, the information that people need to ask questions about and make decisions using, and then you also need the presence of the metadata, which we now call context. And the ways that that has evolved over time, the tooling that exists for that, is evolving in response to what the consumers of this information expect to be present and how they will be using that information. Just as an example, as business intelligence systems started to build on top of the semantic layer and utilize the metadata to make it easy for business users to explore data in an interface, then you needed ways of authoring that context so that it could be exposed to the end users of those business intelligence systems so that they could discover the answers to the questions that they had in a way that was user-friendly for them. Now, agents are going through not a front-end user interface, but a back-end set of documentation and APIs, etc., that allow them to gather the context that they need on the fly using information about the schema, using little notes that have been written in the semantic layer, using additional context that’s layered on top that tells the agent where to go to query what information. And so the role of the data team is somewhat the same in that you’re still authoring the context. It’s still your responsibility to understand the interface of the business domain knowledge and the technical systems knowledge so that you can embed the business domain context in those technical systems. But because the consumption points have changed, you need to author it in a different way. I think the argument on the part of dbt and on the part of Hex is that data teams are very well-positioned to do this because you’ve been doing a variation of this before. But I would say that the systems from which context will be exposed to agents are contested terrain. There are a lot of functions inside of the organization that want either partial or complete ownership over that surface. And so as a data team, it’s really about how we can think about the context that’s needed purely to accomplish our ends, which is to give the business the knowledge that it needs, the insights that it needs, the information arrayed in the correct format that it needs to answer questions and make decisions and take actions in ways that are beneficial to the business overall. And increasingly the avenue through which that needs to happen is via the agent as intermediary. So that is the fundamental change to the role of the data team.
Jason Bradwell: As the skills the data team needs shift towards this kind of more context and semantic work, where does that gap in your opinion start to actually hurt enough that a company has to do something about it, and when does “do something” mean hiring for it internally rather than just bringing in outside help?
Ross Katz: Yeah. I think that it depends on the organization, but many organizations are just going to experiment with connecting their raw warehouse and the schema information directly to AI agents and then try to answer questions with it, because that requires no effort whatsoever. But the smart organizations are going to put the semantic information that they are already using for business intelligence into the hands of AI agents and evaluate how well those agents are doing against the known questions that people ask. And so if you have a data team that is already conducting evaluations on the quality of the responses that AI agents are giving to questions that are in a gold set of expected questions and expected responses, then you’re already in a good place. But in practice, even if you’re doing that, the surface of potential questions that might be asked is always greater than the evaluation set that you’re using. The evaluation set is really just meant to give you directional information, but the minute you use that evaluation set to optimize your semantic layer, you have overfitted to your evaluation set. You are an AI organization that is using a benchmark that no longer captures the scope of real-world work that exists out there. So the sort of thing that I expect to happen that will increase the urgency of building the data teams up to do the platform work that is needed to support truly agentic analytics at scale is: two people are asked a question for the same number, they both ask their agents for that number, and they both come back with different numbers. And neither of them can describe how their agent came up with that number. And so somebody who was relying on those numbers is sitting there wide-eyed like, we have a big problem here. We used to feel like we knew what was happening in our internal and external environment and we no longer know. The second thing that’s going to happen is that somebody at the executive level is going to make a decision based on a confidently wrong answer that they got from an agent. It is a testament to reinforcement learning from human feedback that these agents are excellent at giving us things that sound right. That is the thing they are best at. Whether they are actually right or wrong, the thing almost always sounds right. And so anybody who even for a moment outsources their thought to AI and then decides to confidently make a decision based on wrong information, that’s going to light a fire under companies to invest in it. The other things are like the compute bill starts to go up. They notice that the agents are spending way too much money on getting answers to questions that we used to get for the cost per seat of our business intelligence tool. There’s a regulated, audited, or externally reported metric that depends on an AI-produced number. And people are looking at each other like, who’s responsible for this number? Who’s going to sign on the dotted line here? And nobody is willing to sign because the work hasn’t been done. The person who has the business logic all up in their head leaves the organization and now nobody knows how to get the answers to any questions. These are the sorts of things that I would expect to happen that are going to cause companies to wake up and say, “Oh yeah, we really need to invest in these data foundations and these data capabilities in order to enable our AI agents to help us to accomplish our goals.” It’ll happen to different businesses at different times, because ultimately it depends on how costly an error is. If you’re a solopreneur and you’re flying by the seat of your pants anyway and you’re getting analytics that you never had before, even if it’s confidently wrong, as a solopreneur, having confidence in something that’s wrong is actually a strength. But at the moment when you have a real operating concern with lots of people who have responsibility for the success of that operating concern, and business intelligence has become everybody has their own agent and we’re all throwing agents at the problem, you’re going to realize really quickly that it’s just not the right path.
Jason Bradwell: When a client’s deciding whether to build this context layer with their existing team, whether they are hiring for it, or whether they should bring someone in from the outside, what’s actually driving that decision in practice? Does it depend on the stage of the business? Talk me through some of the levers that may be pulled internally to determine whether we’re building, buying, hiring.
Ross Katz: Yeah. I think that it depends on the organization, but many organizations are just going to… oh, sorry. I’ll start by talking about why you hire an external consultant rather than just building an internal team. Roughly speaking, there are three constraints that companies face that cause them to hire an external consultant to do data work and building context layers. There’s the knowledge constraint, which is, nobody here has done this before. You can hire somebody to do it, but you’ve got the timeline to hire and then you’ve got the timeline for that person to learn the business and what needs to be done, and given the fast-moving evolution of this technology, they may or may not be able to accomplish your goals, and it’s going to be on a much longer time horizon than an external team can come in and do it. There’s the capacity constraint. You’ve got just run-the-business type analytical work that has to get done with your existing team, and this context layer is roughly a one-time build-out with maintenance over time, and so you need to just scale up temporarily to get the thing done, plus you get additional knowledge and expertise from people who have done this externally. And then there’s the credibility constraint, which is that maybe your team knows how to do it, has the capacity, but for some political or bureaucratic reason, the team does not currently have the credibility to build this out or to convince the executive team that this is the right thing to do. And so the external consultant comes in and overcomes the credibility constraint and makes the argument more strongly and more credibly than you could given the forces you face internally. So those are why people hire external data teams generally. But specifically build versus buy versus contract out, I would say that it depends on the nature of the work that’s being done by the agentic AI platform that you’re developing. If you know exactly what this platform needs to do and it’s not going to change over time and it’s fairly generic, then you should just buy a platform that does that for you. You shouldn’t get a SaaS application. There’s no point in building it, somebody’s already built it. You can amortize the cost over a longer period of time, you retain the option to build it in the future if there’s a reason to build it in the future, but ultimately, if the platform capabilities exist and all you have to do is sign on the dotted line and write them a check, and then the capabilities come internal and it’s going to do what you need it to do for time immemorial, there’s no reason to either build it or hire an external team. If it is relatively consistent over time and you know in advance what it’s going to be, but it’s very business specific, and there’s no platform out there that really does what you need it to do, then you’re talking about building an internal team that does it. You can see on the horizon that this is a need that’s going to exist for the very long term and it’s domain-specific expertise that you want to have internally, that you’re going to need your internal data team to be able to understand and develop. Then you’re building that internal data team. Generally speaking, if the work is spiky, if it’s a one-time build for something custom to your business, that’s when you start thinking about an external consultant. Or maybe you don’t know in advance what the requirements of the system are or what the needs are for the business that would drive the value from the agentic AI system. That’s another reason you would bring in an external consultant, because the external consultant, if they’re worth their salt, will be able to ask the right questions, get to the bottom of where the value comes from, and then scope out the specific system requirements that you need in order to drive that value for your business, without all of the additional bells and whistles that you might get from a more expensive platform that does more of what you don’t need and less of what you need. But broadly speaking, that’s how I would think about it.
Jason Bradwell: I want to shift gears a little bit and talk about something that Hex said in one of their articles. They’re making this argument that data teams should be investing more in the infrastructure and less in the one-off analyst work, because from their point of view this kind of ad hoc analysis is becoming disposable while the infrastructure is what is going to ultimately scale. If you were building a data team from scratch today or advising your clients on building a data team today, what would that org chart actually look like in order to set them up for long-term success?
Ross Katz: Yeah. So setting aside the leader, assuming, stipulated, there is a data leader at this organization, because if there isn’t a data leader at the organization then I have questions about who’s overseeing the org chart in question and what the nature of the work is. Broadly speaking, setting aside whether this is a product-focused organization, a marketing-focused organization, a finance-focused organization, which tends to lead to different data team org structures depending on where the value is coming from, what I would think about is using Katie Bauer’s platform versus distribution, and I would add on where the consumption is coming from for the platform and the distribution. Right now your data team is roughly platform for humans and distribution for humans. Platform for humans is the core data modeling, it’s the data warehouse you have in place today, it’s the business intelligence you have in place today. It’s shrinking as a share of the total pie but it’s not going away. You still need people to do that. There’s distribution to humans, which is the relatively senior business-facing people who can exercise judgment over the data that is coming out of the platform and help to guide your human stakeholders to the right answer to their very hard, complex, and ambiguous questions. And we haven’t talked about this yet, but I think it’s just worth hammering on this point: AI makes the production of analytical assets easy. It makes the writing of anything easy. But where that places the bottleneck within a business is being able to exercise judgment about what is being presented to you. Being able to consume it efficiently, understand it, ask the right questions, and exercise judgment about what the relevant parts of what’s being presented to me are, and what doesn’t pass the smell test or is missing an important piece of context or just doesn’t feel right and requires additional validation from a human. So you still need people, in my opinion, within the data team who understand the domain and the data ecosystem deeply and can have those conversations with executive-level stakeholders about what the AI agent is getting right and what the AI agent is getting wrong. When the executive stakeholder asks a question of AI and something isn’t feeling right to them, there needs to be somebody they can Slack. And that person should be on your team, because that is an important source of demand signal for you about what people are asking and what the critical decisions are that they’re trying to make, and that gets fed back into the platform team, and it also gets fed into the distribution team as well. So, roughly speaking, you need platform for humans, distribution for humans, but then you need platform for agents as well, which is the net new aspect of the work that the data team needs to do. Yes, there’s semantic modeling and metric definitions that get to ride on the back of the core BI that you have in place today, but then there’s also: okay, what are the other types of context we need to have available and how do we expose them to the agent? Is it through an MCP server? Is the MCP server already embedded in our data warehouse? How does that fit into the AI platform that we’re using already or models that we’re using already? How are we handling access control? Is it access control to the user? Is it access control via service account to the agent? Do we have monitoring and observability of the agentic platform? All of the governance things we talked about in our last episode with Dave Yaffe. And do we have evaluation suites in place, and are we running them, and are we expanding them over time and holding out a test set so that we know whether we’re overfitting to our evaluation set or not? That is either one or a set of full-time jobs for organizations of a certain scale that needs to exist today. And then, roughly speaking, this could be another person on the team, it might even be the same person as distribution to humans, but distribution to agents. One of the things that’s happened in this new world is that all of your demand signals are gone for the most part. As a data team, you used to get people asking you questions rather than asking questions of AI. You used to get tickets submitted for reports that people wanted or new apps that people wanted to develop. But now everybody is doing it themselves. And if you don’t have that monitoring and observability, if you’re not talking to the users of AI who are accessing your data platform, then you’re flying blind with regard to how you maintain that platform or, strategically, what the roadmap for that platform is in order to drive value in the way that your business drives value. So there needs to be some thought about what the distribution role for agents looks like in this new world. Maybe it’s somebody who’s going out and interviewing people who are using AI. They’re sitting next to them: “What questions are you asking?” They’re looking over the shoulder at what the answers are. “Oh, this is wrong. Okay, that’s not good. I need to flag that for the platform team.” They’re focusing on adoption, quality, and reconciliation: we’ve got multiple sources of truth for questions that we’ve never been asked about before, and we need to make sure that that gets into our enhancement queue for the data model. And as we onboard new source systems, who’s going to want to ask questions about them and how does that impact the surface area that we need to cover from an agentic perspective? So, broadly speaking, that’s how I would think about what a data team looks like in this new world. Roughly speaking, I don’t think the titles are going to change that much. I still think you’re going to have people at an analyst level, you’re going to have analytics engineers who might be titled context engineers five years from now, but it doesn’t really matter. They’re still doing essentially the same job. And you’re still going to have leadership roles who have to oversee the entire data ecosystem and make sure that, whether it’s platform or distribution, the value that the company gets from data is being achieved.
Jason Bradwell: What you said a little bit earlier, which was around being able to apply judgment to what you are getting out of these systems being a fundamental skill. And I wonder if that is at risk in the context of organizations that are trying to go faster and do more with less, you know? My observation is that you’ve got all these businesses who are leaning into AI. AI is the big unlock. We remove all these people, hire less people, we can go faster, we can do more because we’re now empowered to do so. And it’s not like you just get on this hamster wheel, you know? And it’s like no minute. You need to get more stuff done with the same amount of time that you had previously. And so where is the time for judgment? I think sometimes, unquestioning, do you actually have leaders in organizations who, yes, are empowered to make decisions, but also are not feeling the pressure of, well, now I’ve got to fill all this extra time I have, and so I’m going to sacrifice the judgment because it is just so easy to get something and ship something that I can kind of forgo that? So I don’t know. It’s just something I’ve been thinking a lot about in our own business recently. When do we apply the judgment, you know?
Ross Katz: Yeah, it’s a tool that allows you to cut corners in a variety of ways. The way I think about it is you can get B or C-level work in almost no time. So if you’re the sort of person who didn’t want to show up to any classes and just go about your day and get the passing grade, and you exist in an organization and ecosystem that’s fine with passing grades, then yeah, the incentives are there to cut corners. But really, my viewpoint on AI is it allows you to do A-plus work with, call it 75% of the effort or 80% of the effort. And that’s where I try to push myself: my standard for done has gone up, not down. And I think that depending on the incentives within a given company, that might not be the case in some companies. And I also think that there’s judgment that needs to be applied for what “good enough” constitutes in a variety of different places, because sometimes a slide deck that looks like it was generated by AI but conveys the message you want to convey is good enough, you know? And you got there much faster than you would have otherwise. But within the context of the analytics ecosystem, documentation that was AI-generated as a one-off and then never reviewed by somebody who has both the domain expertise and the technical expertise about what’s happening, and then gets used a hundred times a day by executives at your organization, is a disaster waiting to happen. It’s just a landmine. And this is one of the things that’s really challenging, especially for data people who are in the early stages of their career: you can either be in an anxiety spiral where everything is important and you’re detail-oriented to the point where everything has to be perfect or it’s never done, and then you’re not moving at the speed of the organization. Or you’re cutting the wrong corner and you’re laying a landmine that’s going to blow up in your face six months from now. And so that is part of where leadership comes in: setting the standard for what constitutes “done” and imposing the judgment on what the data team is producing, to make sure that there are no landmines and that what we’re putting out there, whether it’s being reviewed by agents or being reviewed by humans, is to the standard of quality that we need it to be, given the importance of the artifacts or models or whatever we’re developing, and the value that ultimately gets driven by those artifacts inside the organization.
Jason Bradwell: 12 months from now, do you see there being a role in the data team that either won’t exist, or one that doesn’t exist yet that will?
Ross Katz: Analyst is an interesting title because when I started as an analyst, analyst meant Excel jockey. Whenever somebody needed to manipulate Excel, you’d go talk to Ross and Ross would manipulate the Excel for you and get you the output that you wanted. And that was not a fun time to be an analyst. I would say that nowadays, junior analytics engineer would be the way that I would classify an analyst. People are still going to be called analysts. We still on our team call them analysts. But maybe it’s junior analytics engineer plus business intelligence report development. But our advice, and I think the way the ecosystem will go over time, is that these tools accelerate the amount of work that can get done so much that even if it requires judgment and even if quality suffers on the incremental item that gets done, the throughput is so high that you’re not going to stop doing it. I think code-based BI is going to be the future of BI just because code is so easy to generate. The amount of reports that you can produce is astronomical if you’re producing them via code versus using a drag-and-drop interface in the current environment. So what I would say is that the analyst role will be increasingly code-based. It was already increasingly SQL-based versus drag-and-drop-interface-based. So I view this as an extension of that. The analyst is going to be someone who writes code, but they’re not able to exercise the judgment about the domain or about the data ecosystem that someone who is given the title of analytics engineer, or machine learning engineer, should be able to exercise within a given business ecosystem.
Jason Bradwell: So final question before we wrap this up, Ross. You’ve got these good analysts. They carry a lot of tacit knowledge. There are these seasonal quirks that everyone just knows about, something that throws the year-on-year numbers off every year, but they never actually get written down anywhere. So if context engineering is all about making that explicit, is that something that’s teachable to an existing analyst or is it a genuinely different kind of hire?
Ross Katz: I would say that it is teachable to an existing analyst or anyone who you had writing dbt YAML configurations previously. If their job was to document the business logic for the report previously, then they should be able to document the business logic for the report now. It’s rare that the analyst was purely just someone who generated SQL. That world was already in the past. I would say that the integration of business domain knowledge is already an expectation on the part of analysts. But when we talk about tacit knowledge, I think we should just ground ourselves in what we mean by tacit knowledge. There’s what you were talking about: there’s a fact about the past that hasn’t been written down. It’s just in somebody’s head. There was an acquisition in this year that makes revenue numbers look all weird. The fiscal calendar affects revenue in this way during this month. These are things that ideally would be built directly into the semantic model so that the calculation errors go away. So that’s the kind of thing that I would expect analysts to already account for in the work that they’re doing right now. Then, when we talk about tacit knowledge, sometimes we mean something more like pattern recognition, where it’s like, this number looks wrong. You can’t really tell someone how you know that a number looks wrong, because it’s based on the complete accumulated experience that you’ve had in your life up to that point, including your experience at that organization. And so, broadly speaking, I would expect someone who’s more tenured to be better at pattern recognition than someone who’s less tenured, although somebody who’s really good at pattern recognition can generally pick it up pretty fast. They have some shortcuts that they take, heuristics that they use that allow them to pattern recognize. Hypothetically AI could be taught things like that, but I expect pattern recognition to be, broadly speaking, outside the realm of what AI can do for the foreseeable future. And then lastly there’s the judgment about what matters. This is the thing that executives are really good at and data people have varying degrees of quality in. What is the question about the business that is worth asking? What is the answer that we can act on? When would I override the answer that I got from the data or the agent or the analyst? There’s no real documentation to be written about this either. It’s just tacit knowledge combined with skill, business acumen. This is what makes you good at your job as an executive or business person or data person. It’s that secret sauce that anyone who’s in the business world tries to develop over time, and some of us have more of a talent for it and some of us don’t. But hard work also cures it. So I would expect that analysts can be asked to handle the tacit knowledge problem at the level of unwritten facts. I would expect data leaders to be able to handle the problem at the level of pattern recognition. I would expect a really good data leader to be able to answer the question that requires judgment about what matters to the business. But ultimately, if you’re going to write something down, the question is inevitably where? Because not everything can be documented at the level of a column or the level of a table. There’s some documentation that needs to be documented at the level of the business as a whole, or a specific customer entity, or the way that deliveries work in this business. The problem of how you organize the knowledge inside of your organization so that you know where to write down the information that can be written down is an unsolved problem. And I don’t expect an analyst to know the answer to that. But what I do see is that, for example, MotherDuck already allows you to use dbt’s persist docs to push column-level and table-level information into the warehouse so that the MotherDuck MCP can query that information directly, using table and column level information. And they just released MotherDuck Guides, which are roughly organization-level documentation that you can put in the warehouse. And so you could imagine: how do you organize these guides? You could have a guide for which data source to go to when you want to answer which question. This is the guide for customer-specific outlier information about how each customer reports on what is needed. The way that those guides are organized is going to allow you to know, this is where we put information like that. And if you’re really good at having that organizational sense of how agents need to be able to access the information about your business, then we now have an avenue, at least within the MotherDuck ecosystem, and I expect similar features are coming within Databricks, Snowflake, BigQuery, where you can tell the analyst, “Oh, that goes in the customer guide.” And so that’s how I expect things to evolve.
Jason Bradwell: Here we are again, Ross, another week, another episode of Eventual Consistency, another “What We’re Watching” segment. So what’s been in your eyes over the last few weeks?
Ross Katz: One of the ideas that everybody seems to be batting around, that I’ve been spending a decent amount of time chewing on, is this idea of who AI helps. Does it help the newbie to become mediocre? Or does it help the expert to become a super person? And looking at the research, my current viewpoint on this is you have to think about whether the thing that you’re doing is very well defined or is it open-ended. If it’s very well defined and it’s been done a million times, then someone who is relatively inexperienced is going to get a lot of gain from using AI to do that thing. If you’re an expert and you’re doing something that’s well defined, then what you’re probably getting is a reasonable increase in speed, but you’re also sacrificing a little bit in the quality that you would get, because if you were sitting there thinking about it, you would be able to do it better than AI would do it for you. But if it’s well defined then maybe you don’t care so much about investing your expert attention in that very well-defined thing, because that’s not where you add value. Now if you’re talking about open-ended type tasks, if you look at this study of Kenyan business people that Otis et al. did, what you see is that for open-ended, entrepreneurial-type problems, people who are inexperienced are like, hmm, they lose. How much they lose is open to question, but the way I would think about it is open-ended tasks are like iterative games. And so the worse your judgment is and the more iterations you have, the more dead in the water you become after bunches and bunches of iterations, because you don’t know what that gap was or where that turn-off was that takes you from landing some place great versus landing some place that’s terrible. And if you miss that turn or that set of turns, then you’re basically dead in the water. And this is my experience with working with AI outside of areas where I have domain expertise as well. Now if you’re an expert and you’re doing open-ended exploration, then that’s where I would expect the largest absolute gain. The superusers of AI, the people who I see who are accelerated the most, are people who are already experts in a given domain or set of domains, and the Otis et al. research supports this, that there’s this huge gain that comes from working on open-ended tasks. And this is how I reconcile the arguments. The argument is, who does it help, who does it hurt? Well, if the task is well defined it really helps novices, and if the task is open-ended then it really helps experts. And if the task is well defined then experts take a penalty, but it’s a penalty they may not care about in the real world. And if the task is open-ended then novices take a penalty, and it basically means that novices cannot use AI to do open-ended things that they couldn’t do before. So if I were thinking about where I would invest my skill development capabilities as a human in the age of AI, I would be thinking about how do I make myself better at managing ambiguity and at completing open-ended things, like doing more creative projects that require me to exercise my judgment and make mistakes and do things that are open-ended, because over time I expect that doing these open-ended problems will make you better at the open-ended things and allow you to compensate more effectively for the acceleration that you get from AI for well-defined things.
Jason: What a great episode again. So the data team still has a platform job and a distribution job, but now it has both of those for agents as well as humans. And that second half is the work that nobody has started for yet. The warning signs to watch for: two people asking for the same number and getting two different answers; a reported metric that nobody will sign off on; compute costs climbing for questions that you used to answer with a BI tool; and Ross’s line on standards: AI should push your bar for “done” up, not down. If you want to talk to the CorrDyn team about any of this, head to corrdyn.com, that’s C-O-R-R-D-Y-N dot com. Thanks for listening and we’ll see you on the next one.
