Listen on
Overview
Prefect bought Dagster, and the obvious reading is that a slower grower got folded into a faster one. David Yaffe’s reading is that orchestration was never the asset. Airflow is free, Astronomer sells the managed version, and both Snowflake and Databricks have every reason to commoditize orchestration because they make their money on the compute underneath it. What Prefect owns is FastMCP, the open-source framework Anthropic adopted as the way to build MCP servers with Claude, and Horizon, the hosting, catalog, and permissions layer going on top of it. The combined orchestration business generates the cash; the agent governance product is the bet.
David Yaffe is co-founder and CEO of Estuary, which unifies change data capture, batch, and streaming into one system for moving data in real time. He was previously COO of LiveRamp, having sold his own company Arbor to LiveRamp in 2016, and before that led product at Invite Media, which Google acquired and turned into DoubleClick Bid Manager. He returns to the show after an earlier conversation on efficient data streaming.
In this episode of Eventual Consistency, Ross Katz and David work through what Ross calls the acqui-pivot: keeping a cash-generating orchestration business intact long enough to fund an agentic ambition, while the buyer for that ambition shifts from the data engineer to IT and the CIO. They get into why a well-documented CLI can beat an MCP server on token cost, why an agent should be treated like a new graduate on day one, what time-limited per-task permissions look like in practice, and what a team running a large set of Dagster DAGs should expect over the next year.
Key Takeaways
Orchestration was never the prize
Scheduling pipelines is a solved problem with a free incumbent. Airflow sits on the open-source side, Astronomer sells the platform version, and the warehouses bundle their own orchestration because they profit from the compute it triggers rather than the scheduling itself. Yaffe’s point is that a company worth $80 billion, or Databricks at something closer to $190 billion, can fund an orchestration layer good enough to erase the reason anyone buys a standalone one. Governance is where a vendor can still get lock-in, which is why Prefect taking ownership of FastMCP looks less like a side project and more like the plan.
The acqui-pivot: buy the cash flow, fund the pivot
Dagster’s growth had flattened, which left a down round or a sale, and it sold for an undisclosed sum with only part of the team retained. Prefect gets a large open-source community as top-of-funnel and an orchestration business that is unexciting but cash-generative. Announcing that the Dagster brand stays intact is the confirmation: the operating company has to keep running long enough for agentic revenue to arrive, and Yaffe doubts anyone expects it to run as a standalone business for years. Agentic cash will be slower to appear while buyers work out who funds it internally.
The buyer changes, and the two buyers don’t speak the same language
Prefect and Dagster both sold to VPs of engineering and data engineers who needed a friendlier Airflow. Agent permissions and observability are an IT purchase, or a CIO one, made by someone reasoning about liability across the whole company rather than one team’s pipelines. Yaffe expects that to be genuinely difficult, because the economic buyer and the functional buyer have to be brought into the same conversation. Ross frames the consequence plainly: a pivot to a new buyer and a new use case with only marginal technological overlap with the previous business.
Documented CLIs beat MCP servers on token cost
The complaint you find on the first page of search results about MCP is that a server burns a lot of tokens grinding through information to answer a simple question. Yaffe’s team went the other way and settled on standard documentation for a CLI. If the CLI is documented well enough for a novice human, an agent will probably manage one help query and then work efficiently, with no MCP server to maintain and a side benefit for any human users who show up later. Claude skills occupy the middle ground, giving an agent more direction than a general-purpose server without the same overhead.
Treat an agent like a new graduate on day one
Yaffe’s framing for anyone building agent access to data: the agent is smart and has no idea where anything is. If your schemas, data model, and documentation are not clear enough for a new hire to follow, the agent will not work it out either, and it will spend a lot of money failing. That inverts the usual complaint about agents and data. The most valuable thing a team can do is the documentation and metadata hygiene it has been deferring, because that is what makes an agent capable of understanding the company’s own historic data.
Time-limited, fine-grained access is the new primitive
The largest new API Estuary built for its agentic features grants specific, time-limited access to an agent for a single task. Give an agent less permission than a human you already trust, and expire it quickly in case something goes wrong. Ross adds the adjacent controls people are now inventing on the fly: caps on tokens spent, caps on wall-clock time, caps on scope, all of them attempts to bound a process that can otherwise loop indefinitely. Where those controls are supposed to live is contested, and a lot of money is moving toward the answer.
Regulated industries need a chain of custody before they need anything else
Yaffe’s clearest case for a governance vendor is liability transfer. In regulated industries every decision and change has to be tracked by law, and the reason a company pays an outside vendor is so that the vendor carries the obligation. That is how it already works with humans, and he expects it to work the same way with agents. His assessment of where the tooling stands is blunt: nobody feels confident they know what their agents are doing today, and the solution is still some way off.
Related: CorrDyn builds the data engineering foundations and AI strategy that decide whether agent access to your data produces reproducible answers or expensive guesses. Also on this thread: What MCP Means for Enterprise Data Strategy and Similar Keynote, Different Platforms: Snowflake vs. Databricks.
Full Transcript
Jason: Two of the biggest names in workflow orchestration just became one company. In July, Prefect announced that it was acquiring Dagster Labs, the two most widely used successors to Apache Airflow. In a deal that most people are reading as straightforward consolidation, a slower grower getting folded into a faster one. That’s the easy read, but it’s probably not the right one. Prefect built FastMCP, the open source framework Anthropic later adopted as the standard way to build MCP servers with Claude. And it’s already layering a governance and permissions product called Horizon on top of it. Add Dagster’s asset-based approach to defining and verifying outcomes, and what actually emerges is a company positioning itself to own how AI agents are governed and permissioned, not just how data pipelines get scheduled. To dig into what’s real in that story and what’s just positioning, Ross Katz sits down with David Yaffe, co-founder and CEO of Estuary, the real-time data integration platform, in this week’s episode. They get into why this deal makes more strategic sense than it first appears, why the buyer for agent governance tooling is shifting away from the data team and towards IT and the CIO, the case for documentation over MCP servers when you’re building for agents, and where they’d expect the next wave of consolidation and opportunity across the data stack to land. I’m Jason, this is Eventual Consistency from CorrDyn.
Ross Katz: Dave Yaffe, welcome to Eventual Consistency.
David Yaffe: Hey Ross, good to see you again.
Ross Katz: Good to see you too. We’re talking today about Prefect buying Dagster, which happened a couple of weeks ago. So you’ve got two of the best-known names in orchestration becoming one company. And so, I’m just interested in when you saw this come across the wire what was your gut? What was the first thing you thought of?
David Yaffe: Yeah, I think, well the first thing I thought was the fact that they were both this 200… 2021 raising a bunch of money cycle. The vintage was interesting because we’ve seen, there was always this prediction that vintage of startup companies would start to slowly fade away. There was just too much money raised over time. And I think my initial thought was that they were both just starting to do that. It was a facet of the times that they were in and all of that. But I think over the time since then, I have thought about it a little bit more and it’s a little bit more of a natural merger than I originally thought about mostly because of how the two products have evolved over the past few years and really like the past year. And some other things that I’ve heard about both companies just through the grapevines and stuff. But I think this is the same buyer VPs of engineering that are buying a product, basically a similar concept of product, originally when they came out, they’re both basically better alternatives for Airflow, a little bit more dev-friendly. And so it’s like a bit of a nothing but at the same time one that does make sense.
Ross Katz: So can you unpack that a little bit? What so what about this do you think makes sense other than obviously from the Prefect’s perspective, they’re like consolidating the buyer? So they have more access to more of the buyers. Anybody who’s buying an orchestration tool has at some point been in the pipeline for either Dagster or Prefect I would expect. But are there other reasons why you think this makes sense strategically?
David Yaffe: Yeah one of the biggest ones is just I’ve thought that the moves that Prefect has made in the past few months or longer than that, have made a lot of sense. Prefect has FastMCP and FastMCP seems like it’s been a pretty good take on where orchestration is going in the more agentic world. So that makes sense. Dagster’s been a little bit more on the staying the course side. Has a great open source community. The open source community makes a ton of sense why wouldn’t you want a new place to get top-of-funnel users from if you’re Prefect? And I’ve also just heard some rumblings that growth on the Dagster side has been a little bit flat and like not quite as fast as they wanted it to go. So all of that together feels like it’s a good match for everyone that’s involved that’s at least not a customer of the platforms but a stockholder or in either the platforms.
Ross Katz: No, that aligns with what I’ve seen as well. And I think the there a couple things come to mind from a strategic perspective. You know it seems like orchestration as a portion of the data stack was just not a place that a lot of like, from a paying customer perspective, was not a place that a lot of value was going to because as you mentioned, you have Airflow on the open-the open source side. You have Astronomer, the platform version of Airflow out there and available. But then also, you have you have Databricks and you have Snowflake and you have all of the public clouds offering competing versions for data movement and they’re charging for compute and so they’re more than willing to commoditize their complement in order to make that business disappear. What do you think about that idea?
David Yaffe: Yeah, I- absolutely. It’s a super interesting space. The I think the entire stack, right we have to look at the whole stack from first principles and agentic workflows, LLMs, they all make it so possible and so I wouldn’t say easy but at least accessible to be able to bring new workflows into a product. So naturally, if you’re a company worth eighty billion dollars like Snowflake, I’m probably even on the low side now. Snowflake or Databricks, Databricks is what, 190 or something now? Then you’re going to be able to invest enough to, in a modern world, have a pretty successful orchestration layer. So it’s going to be built into the tools. They’re probably going to be pretty decent. The thing that you’re able to get more lock-in on as a vendor is going to be something that’s a little bit more novel. Maybe the governance side of things. Getting your feet into the governance side of things is going to really like get you in there for the long term and make you a lot more sticky than you would otherwise. So if you can do that, well when agents and agentic workflows are coming out soon. I think that FastMCP project ownership by Prefect was just like a wonderful idea. Like it really jumpstarted their capability to get into it for like the modern age and solidified their groundwork to be able to take control of it.
Ross Katz: Yeah it’s funny because on its surface, it doesn’t exactly make sense to me why Prefect would have been the company to launch FastMCP. But as the ecosystem has evolved, you see how at one of the best use cases for agents is on the analytics side, and that to a large degree running agents, governing agents interacting with agents there’s a lot of orchestration problems in there. Am I thinking about that right, or is it really just that their CEO just right place, right time with the right Python framework getting out there?
David Yaffe: No, I think they were like really ahead of that trend and they figured it out where they needed to be going in this new world before a lot of other people even caught on. Like, I think every company has the problem right now of okay, so we want to empower our users to be able to run agents. How do we do that? Like, how do I make sure that my users are not going to delete our entire codebase when the non-technical users are given access to do something with an agent? That’s- I think there’s probably not one leader that has not thought about that recently. Like, every single one has. So what’s the answer? The answer is, okay, so we need an environment that these things run in. I don’t naturally think about scheduling, but I naturally do think about the capability to run them, and mostly around permissions and governance and understanding what they’re doing observability, all those things together really are the biggest things that I think about. We actually have been building an agentic product into our platform, so it’s the capability to basically do event-driven LLM-based transformations. So someone comes to your website, you get a ping from HubSpot, HubSpot tells you this is the domain of the user that came to your website, and then you have a little BDR agent that goes to town enriching the record and trying to send an email faster than that person can block you. And so that’s a workflow that we’ve been thinking about, we’ve just been like, okay, how do we make sure that the permission that the agent’s given is locked down to exactly what you want at that time, and all of that is a pretty complex problem. I think the one question is like, is MCP the right framework? But that’s a much deeper question.
Ross Katz: Can you unpack that question? Like what so what are the alternatives from the perspective of the hard parts of this to MCP? Because I’ve heard the MCP versus CLI debate James and I were on this podcast like, I don’t know, a couple months ago talking about MCP versus CLI, but I’m interested from your perspective as a builder in this space how do you think about some of the problems of MCP and what like, potential solutions might look like?
David Yaffe: And MCP, the biggest problem that you’ll hear about if you write one Google query is that it will spend a lot of tokens getting an answer. Like, it’ll start up and have to crunch through a whole bunch of information to get the answer for a very simple simple task. Like how do I actually efficiently crawl this website, this API and so there’s a lot of options. Like you can build an MCP server which is like a hammer that will solve your problem, or you can get a little bit more detail-oriented like there’s Claude skills and stuff like that will tell an agent how to it’ll give the agent the proper types of skills that it can use to then be a little bit more directed when solving the problem. A third level deeper is the, as you said, the CLI debate. And so I think the CLI solution is actually a really interesting one. What it forces is great documentation of your CLI, right? Like if your CLI is documented well enough for a novice human to be able to use it, then an agent’s probably going to have a great time with it and not use too many tokens figuring it out. Maybe it’ll do like one help query and figure out how to use your CLI and then direct itself to answer your question very quickly and efficiently and you don’t actually have to maintain an MCP. It’s a little bit more efficient and it also helps the human users if you ever have any more in the future. It feels like a really nice way to solve the problem and just be more standalone. So like that’s the way we landed for this type of problem versus MCP. It’s just standard documentation.
Ross Katz: No, and really the question of like, how do you partition that documentation? How do you expose that documentation to the agent in a way that makes it easily accessible to that agent in whatever environment or sandbox that agent happens to be in? These are hard problems that are getting hammered out over time. But ultimately, I- it seems like you’re right, that it’s really just making sure that the documentation is great. And if the documentation is great and it’s really well-structured so that you don’t have to read two hundred pages in order to understand how it works, then you’re, then you’re on the path toward making it fly.
David Yaffe: Like think of an agent as a new grad that’s coming in joining your company right out of school. They’re smart they just have no idea where anything is or what anything is. And if you don’t have documentation around your schemas and your data model and all that stuff, the agent’s never going to figure it out either. Like it’ll spend a ton of money and maybe never figure it out.
Ross Katz: Yeah. So back to Prefect and FastMCP. You know, I heard their CEO, I- think it’s Jeremiah Lowin, on the MLOps podcast talking about some of the stuff that they’re doing with FastMCP and Horizon agents, their platform that they’re laying on top of it. You know as always they’ve got this, with companies that are built on open source frameworks, they’ve got FastMCP, which is open source. It’s over a million downloads a day. Anthropic has basically said this is the way to build MCP servers with Claude. And then on top of that, Prefect is building this Horizon, which includes like hosting, a catalog, a permissions gateway, and then increasingly it seems like a front-end interface with Prefab, that they just released, which is, like a- DSL, a domain-specific language for React components that they’re trying to compress the tokens for rendering a front end into something that’s really small. And the point that Lowin made was that the vast majority of MCP servers are for internal use and not actually for interfacing with customers. And so like the buyer for it in that world where they’re mostly in where it’s mostly for internal use is to many degrees the same buyer as for data stack components. So I’m interested in what about the idea of the of like the same people who buy orchestration tools being the buyer for your governance stack and your interface stack for agents on top of an MCP server? Do you think that makes sense?
David Yaffe: I think it could. Like at some level they’re probably having to factor that stuff in to an orchestration stack now. But it’s not exactly the buyer I would think of. Like you’d usually think of it as an IT department or something along those lines that’s thinking a little bit more holistically than the data engineer that’s actually scheduling jobs and doing stuff like that. So it feels like you’re needing to get a couple of different personalities involved in this and that might be actually tough to like get them thinking the same thoughts and talking the same language. So I’m curious what you think on that one.
Ross Katz: Yeah so it’s like there’s just a- I’ve been trying to figure out what the right portmanteau is for acquisition and pivot. Like an acqui-pivot. Because the because they’re buying Dagster. To your point earlier, Dagster’s growth is slowing. Their options are limited they’ve basically got try to do a raise in a down round because they have some strategic pivot in mind for themselves or do what they did, which is sell for an undisclosed amount. You know, only a portion of the team is kept. The leadership goes off and does their own thing. And from Prefect’s perspective, like all of the money in the ecosystem seems to be flowing into AI and agentic AI. So for them, they’ve got this new and emerging open source platform that is gained a lot of momentum, that lots of people in the ecosystem are using, and so they want to be able to keep and maintain like the cash generation of the orchestration business, which is it’s not a bad business. It’s just not a good exit for any of their investors to your point about like the 2021- for the 2021 vintage. But the potential to combine forces potentially raise on the back of the momentum that they’re seeing on the agentic vision for Prefect while maintaining like the operating business of orchestration, like strategically speaking, that’s the part that makes sense to me. But in terms of the buyer I think you have it right, which is that the IT department or the CIO is the buyer of the agentic governance system, not your data team or your data engineers or the people who are working on Dagster’s open source platform. So really what you’re doing is you’re pivoting to a new buyer and a new use case with marginal technological overlap between your previous business. But you’re creating like a bigger operating co that throws off more cash, which gives you the platform from which to invest in the agentic AI angle and maybe raise more money for the agentic AI angle. What do you think about that idea?
David Yaffe: Yeah, I think that’s a hundred percent accurate. Like the one of the things that’s interesting about this deal is that they announced that they’re not going to actually do anything with the Dagster brand. They’re going to keep it and it’s all going to be intact and everything like that. That right there shows the writing on the wall. Like they obviously want to keep all the Dagster customers, they want to keep that going, they want to keep that operating co intact so that they can enable the cash flow that’s coming in. I think cash flow from newer agentic projects is probably going to be a little bit longer to take to actually come in as businesses even figure out how to fund them, as they figure out how to get the economic buyer as you mentioned and the functional buyer aligned. They’re going to have to play a song and dance to get that all going together. No, it’s going to be a really interesting time. I don’t believe that they’re actually going to keep Dagster operating as a standalone business in and actually be successful for the next several years at that. That’s probably not the plan though. The plan is probably just keep it going long enough to get the cash that coming in from the agentic side of the business to be the meaningful part and then get rid of it. Like, then you don’t have to even think about the cash that’s coming in from the other side of the business, from the orchestration classic side.
Ross Katz: That sounds right to me. And it’s just so funny because the among the data wonks of the world, there was this whole argument about asset-based tagging and the benefits of Dagster versus like task-based and like in the in the Prefect world. And now they’re under one umbrella, which I think honestly allows the salespeople internal to them to be more honest about where each use case makes more sense. But then I take very seriously what you’re saying, which is that maintaining two independent platforms like that internally in addition to trying to chase this agentic AI angle, like it seems like a lot of overhead to invest in a business that’s fundamentally not growing. So I want to turn the question back to the from a buyer’s perspective or from people who are already using Dagster right now. Like, how would you be thinking if you were a company that was that had a big set of dags built on Dagster right now?
David Yaffe: First off, didn’t Prefect actually add the concept of asset-based tagging recently? Wasn’t that something that yeah. So that it’s not even it was almost an admission that oh, maybe there’s some use cases for this thing. Right. So that’s telling in its own. I’d imagine that there’s feature convergence of the two products. And so even if it wasn’t for the agentic aspect of it, we’d be looking at a world in which they were working on making Prefect the end-all product where once all that feature convergence happens, you get your contract and you can move over from Dagster, which is end-of-lifing in a year. But don’t worry that we’ll have API functionality and all the other things that you need on the other side as well. So I think realistically, no one keeps two products forever. You might keep it for a while, but what’s going to slowly and maybe not that slowly happen is that the product is probably going to have its roadmap cut a little bit. You’re not going to get all the same updates in both places. You’re not going to see the equivalent type of tagging happen on the that probably was planned on the Dagster side. And so like over time that’s just going to happen. And then you’re probably going to see stuff like the sales team have incentives to kick you over and just the longer-term marginal end-of-life features that come into play with any of these acquisitions will start happening. So I think that’s guaranteed to happen. It’s- I don’t think that it’s a huge risk for anyone who’s actually on it. Like it’s probably, there’s a core incentive that’s really helping you if you’re on Dagster right now. And that core incentive is that Dagster is a big open source community, it’s got a lot of users, it’s got a huge top-of-funnel for Prefect now that they want to maintain. And until that’s, until they have a way to make up for that, it’s going to continue. It’s just probably not going to get the same type of velocity and roadmap that it was.
Ross Katz: Yeah. No, I think that makes sense. And also that must bring some people peace of mind. I think the there’s the carrot and stick aspect of moving people from one platform to another once an acquisition like this happens, and you just hope there’s more carrots than sticks, where it’s like if you want this added functionality then why don’t you come over to Prefect, but if the, but then then the sticks start to come out of the end-of-life or certain features are no longer supported or, etc. Etc. If you don’t move to Prefect then you can’t have the parity in terms of functionality that you wanted previously.
David Yaffe: And that’s definitely the case. Like, and that’s all driven by competition. How much competition is in the market? Is there a viable alternative aside from the two, which there probably is? You have Astronomer and you have Orchestra and other new entrants and stuff like that. So you can’t have too much stick in a world like that. You have to work with carrot. So that’s probably more likely that’s going to happen. This isn’t Google as the acquirer where it’s like, oh, you have six weeks and everything’s shut down. It’s a little different.
Ross Katz: Right. And the life of the company is not on the line for the success of this acquisition. It whereas in the case of Prefect the life of the company is a- is on the line for making sure that they’re able to realize the benefit of this acquisition in order to throw off the cash needed to complete the acqui-pivot that they’re undergoing if they’re, if they’re in fact doing that. There’s been a lot of consolidation in the data space over the last few years. So you’ve got Fivetran and dbt and SQLMesh and you’ve got IBM and Confluent, etc. Etc. Databricks on a spate of acquisitions. You’ve got Snowflake making acquisitions. I’m interested in your perspective on where you think consolidation in the data space is most likely to happen versus where new and emerging tools have the biggest opportunity to capture market share.
David Yaffe: That’s a good question. I think this is a really good example of a place that a new and emerging tool got swallowed up by a pre-existing tool to deliver the new the new version of the existing tool. Like, with FastMCP and Prefect. And so I think we’ll see a lot of that, actually. Why we’re going to, there’s a lot of cash in the space at this point. We have multi-billion dollar companies that are all vying for the same dollar. And so, of course, they’re going to be snapping up companies trying to add them to their core suite of products. The interesting will be thing will be to see like how far a new and emerging product can actually get in this current world before getting snapped up. I think once there’s a category the big players are really going to try to make their own entrance into it pretty quickly. That’s one piece. I think getting a little more specific on the answer to your question, I mentioned governance as a- an interesting place. It is a really important place. It’s like a- I think a much more important aspect when you think about the capability for how autonomous things are going to be, for how much observability you’re going to need onto what they’re actually doing. All of that is, it’s going to just take new tooling, new technology. And I think we’re just like basically at the very tip of that. Like there’s basic solutions that help you with some of it, but I don’t think anyone feels warm and fuzzy that they know exactly what’s going to go on with their agents today.
Ross Katz: Yeah. No, I think that makes a lot of sense. And also it people like to talk about agents as people. You the sort of the metaphor we use is like you’re hiring a bunch of agents to do these tasks. And while that metaphor often falls over and I really don’t like it occurred to me that when companies start to scale up from the perspective of the number of people that they had internal, there was an entire suite of tools that came about from Workday to Gusto to Rippling that are all about making it easy to understand and manage and govern the permissions and the access and everything that these individual people are doing within your company. And so as organizations think about having agents that operate within every domain of the business and operate across data boundaries within the business anytime you hire a person, like the question of specifically which data systems should this person have access to and what permissions should they have within that system is like a big and complicated question. And if you go from whatever, hundreds of employees to thousands of agents doing this kind of work, like those governance problems just and observability problems just get more and more out of hand. And so, yeah, I think I think when I think about it that way, it makes sense that would be one of the areas where monetizing this could really be possible.
David Yaffe: And just imagine regulated industries where a chain of custody is important. You have to track every decision, every change that’s made, and legally you have to do that. You just need a solution that’s going to do it for you. You need a company to pay to say that this company is doing it for me so that I’m not liable for it and I’ve put that liability off on another entity. And that’s the way it works now with humans, that’s the way it’ll work in the future with agents. It’s- I think we’re a little bit away from something like that, but we need it.
Ross Katz: And just the long tail of use cases is just so huge that like, and this is what Prefect is running up against with Horizon, it’s what I imagine you’re running up against with your event-driven agentic features that you’re developing. It’s like, how do you give people the right interface to configure the agents and the right controls to make sure that those, that those agents are doing what they say they’re supposed to be doing, but also giving them the observability and the alerting to know when something’s gone off the rails. And as I talk about it’s not all that different than the type of interface that you develop for data movement where you need to understand what data is being moved, when there’s a problem with it making sure that the movement tasks have all of the permissions that they need. Am I thinking about that right, or how would you add to that?
David Yaffe: One of the big pieces of core infrastructure that we’ve added to track this is a much more fine-grained capability to grant specific time-limited access to an agent for a task. And that feels like it’s a pretty important thing. You don’t want to grant too much permission, probably a lot less than you’d grant a human that you have some sort of level of trust for, and you don’t want it to last too long, just in case something happens. Both of those things seem like they’re really important. That was like the one of the biggest things that the biggest new APIs that we ended up building for this type of thing. But there’s a lot more too. Like it does go hand-in-hand with how data processing and data pipelines work where you do need to alert when something fails. At the end of the day, it is a data pipeline. Like it’s a thing that’s augmenting data, probably with new data and so a lot of the corollaries are similar there.
Ross Katz: Yeah. No, and the and what you’re talking about with time-bound agent access, the capacity for agents to potentially run in infinite loops like there is now no like time-boxing that you can do on the tasks that they’re executing. And so the ability to set expense-based limits, I know is something that was discussed in the MLOps podcast, like the ability to box in terms of the amount of tokens they’re allowed to spend, to box in terms of the amount of time they’re allowed to spend, to box in terms of the amount of access they have. And all of the interfaces with this are just being created on the fly. And who ultimately or where ultimately those systems are supposed to live is like a bit contested area that it seems like a lot of money is flowing into. Interested to see how it evolves.
David Yaffe: And can you unify it across multiple different products? Is that possible? Like, can you have a full orchestration layer that does that across multiple things? And we’ll see. It’ll be really interesting to watch it.
Ross Katz: Let’s say we’re back here a year from now, and we’re having this same conversation about, like governing agents, about the combined Prefect and Dagster and what and how things have evolved in the data space. What would you need to see on Prefect’s side in order to know that this acquisition was the right thing for the company?
David Yaffe: I guess there’s two parts to the thesis that you and I just talked about. One part was that there’s enough cash being thrown off by the combined entity, the operating entity, to be able to achieve their agentic vision. And so, they probably already know if that’s the case. Like, they’ve done the math and figured that one out. So they probably can make sure that’s going to happen. I think what they’re really trying to do though is achieve that agentic vision. Like, is, we the last thing that you and I just talked about was whether in an agentic type of world in the future agents are being orchestrated by a centralized system, whether that’s going to exist. Is that realistic? Like, I don’t think as we’re building that into our product, we would reach for something off to the shelf for it. So in some use cases and instances, it probably wouldn’t make sense. But I could also imagine places that it would. If I was just building an analytics project, maybe it would make a lot more sense to reach for something off the shelf and do that. But I think that’s the real question. Like, how what is the workflow for someone who’s reaching for an off-the-shelf component? Like, is there an OAuth of agents. Can someone build that meta-authentication layer that works, and you have SAML and all these things that work across an enterprise and you can turn it off with a- click of a button, and have a management layer and all that stuff? If that happens, then they clearly won, but I’d say anywhere between not going bankrupt in that is probably a win.
Ross Katz: For sure. And also as you’re describing that I can just imagine the long list of very well-funded players who are all running up against that opportunity at the same time. It’s a big opportunity, but there is a ton of competition, and I think a lot of people would describe that as their vision for the next year for their organizations. And is there anything that you would expect to see at the intersection of agents and the data space that you would either be particularly excited about or something that you’re expecting to see?
David Yaffe: So for agents in the data space, I think that there is there’s a lot of promise, and most of the actual pain has been in companies’ historic buildups of their own data sets. Like the thing that would really unlock a lot of value would be the capability for an agent to be able to understand data in a deeper way than it possibly could. Maybe that likely goes back to what I was saying earlier, it’s really about pretending that an agent is a new grad and being able to explain your data the way you would explain it to that new grad. A way that helps with metadata and labeling, like something like that could really be a huge win. Something that forces humans to get better about their hygiene to unlock the power of agents would help the data space the most.
Ross Katz: That makes a lot of sense. And I also know that there’s- like, there are people who are building features in that space too, but once again, it’s a problem of yes, you can see what’s inside of your products or the products that you have access to, but the sprawl of products and workflows across this diverse software ecosystem that businesses operate in pushes back against this unification that we want agents to be able to embody. And also the more unified the ecosystem gets, the harder it is to govern because more things can go wrong. So it’s a constant tug-of-war. So, Dave we’re doing what you’re watching, so what are you watching right now?
David Yaffe: So I’ve actually been reading a lot lately and not watching as much. Watching Wrexham and dumb stuff like that, but a lot about John von Neumann. And that has been a really interesting. There’s a book called The Maniac, which I think is something that I’d recommend for anyone who’s interested in it. And it’s led me down just a path of researching him and Gödel and all the other people from that day that kind of built the foundations of computing that we are using today. So it’s a really interesting time to be thinking about that and how they were testing these fundamental theorems that were proving that computers couldn’t do stuff that computers are actually doing today. And I think that’s pretty cool.
Ross Katz: So the thing that’s on my mind is maybe– tangentially related to what you’re talking about, which is the hand-wringing that’s going about, about the death of literacy in our society and the extent to which no one, apparently except you and people in your cohort are is reading anymore. And I’ve been thinking a lot about what are the skills and competencies that people are going to need to have in AI-enabled organizations moving forward and how does it feel to work in an organization where AI is being used to accelerate the kind of work that we do on a day-to-day basis. And one of the things I notice about AI-accelerated businesses is that writing has become very easy, that more and more text is being produced inside of our businesses than ever before.
David Yaffe: It’s a mixed bag though.
Ross Katz: It’s a mixed bag. It’s definitely a it’s definitely a mixed bag. But-
David Yaffe: Not necessarily high-quality text.
Ross Katz: Absolutely, which actually brings me to my second point, which is that because so much more text is being produced, to varying degrees of quality, it is incumbent on the people whose job is to manage AI agents and interface with each other with these walls of text that we’re sending, to be able to read and digest all of the information that’s being created and create information that is conducive to people reading and understanding what’s happening. Which means to me that, in with the with pronounced as The Atlantic did I- think a week or two ago, that the death of reading we’re actually entering a period where reading comprehension is going to be one of the critical skills that everyone brings to their job, including people in the technology space who want to interface with and manage AI agents primarily in the form of text. So that’s something that’s on my mind and that I’m watching evolve here as we look toward the future. You have any reaction to that?
David Yaffe: Definitely. So that strikes a- chord with me for sure. I have been trying to spread on my team this burden on yourself to proof stuff that you’ve written with AI and to shorten it, to take out all the unnecessary bits, because it’s unfair to the reader. Like, you can’t ship over a wall of text and assume that the reader is going to get anything out of it whatsoever. You really need to think about that. Like, one of those skills that I learned, my earliest skill probably with working at a real company in the real world was how to write an email. Like, it sounds like a really easy skill, but there’s a lot of really bad email writers out there. And getting someone to do what you want with an email is a trick because what you want is to make it as short as possible because people don’t read much. They read two sentences maybe. That’s the exact opposite of what we see today. Telling people to audit what they I still audit everything that I send down because I just don’t think it’s fair to someone. You’re not going to get what you want by just sending them something that’s straight from AI. You’re probably going to get someone who puts what you sent in AI into AI to summarize it and then it will say something different.
Ross Katz: Totally. And I- it just brings up a bunch of thoughts for me about obviously agents can consume a lot of text, but the text needs to be accurate. So it’s okay to have very detailed walls of text and not care about the style of that text if it’s just being passed from agent to agent, but it’s incumbent on you to ensure the accuracy of that wall of text, which to your point comes back to, are you reading and reviewing the things that you’re generating and taking ownership of validating them. Yeah, I’m going to be interested in seeing that evolve because despite people investing in Neuralink, and despite the emergence of models that can generate video or can listen to voice, I expect that because of the density of information that can be conveyed through text, we’re going to be dealing with text for a long time. It’s a big problem.
David Yaffe: Yeah, I’m not looking forward to the amount of text that I’m going to have to read in the next several coming years.
Ross Katz: For sure. All right, well, Dave, thanks so much for joining. This has been great.
David Yaffe: All right, thank you.
Jason: That was a great episode, and if there’s one thread running through that whole conversation, it’s this: the orchestration layer isn’t where the value is anymore. Scheduling a pipeline is a solved problem. What Ross and David kept circling back to is that the real fight, the one that Prefect is clearly positioned for, is over who governs what an agent is allowed to touch, for how long, and who’s accountable when it gets something wrong. The practical takeaway for anyone building with agents today is a simple one: treat them like a smart new hire who’s never seen your systems before. If your documentation, your data model, your schemas aren’t clear enough for a new starter to follow, then an agent will struggle with it too, just more expensively. And that applies just as much to the words that you put in front of people and increasingly agents every day. Writing has never been easier; writing clearly and accurately is quietly becoming one of the most valuable skills in the business. Thanks to David from Estuary for joining Ross on this one. You can find out more about CorrDyn, including the full back catalog of Eventual Consistency, at corrdyn.com. We’ll see you on the next one.
