Listen on
Overview
Most people split AI into two questions: can the model do the task, and will anyone let it run in production. Matthew Sekac, Head of Data & Analytics at Welocalize, argues that split falls apart the moment real customers are involved. For a localization business, a wrong output can cost the account, so there is no room to assert that a system works — it has to prove it, repeatedly, with data that convinces skeptical stakeholders and the team building it. Trust is not bolted on after the capability exists. It is part of what capability means.
In this episode of Eventual Consistency, Ross Katz (Principal and Data Science Lead at CorrDyn) talks with Matthew Sekac about an 18-month agentic-AI rollout built for the coordination work behind thousands of translation and localization projects. Rules-based automation had already handled the common-cause majority of the work; what remained was a long tail of over a million special-cause tasks a year, too varied to encode by hand. One customer’s single program was a 56-page SOP that turned into a 70-row spreadsheet once the team decomposed it from direct observation.
The conversation walks through how Welocalize proved the system before trusting it: an ethnographic study of how project managers actually work, a 30-day shadow test that ran agents against live human decisions before any customer saw the output, a checking agent whose confidence ran backwards from accuracy, and the evals-based discipline of deciding how you will measure success before you build. The through line is a benchmark most AI conversations skip — human error, not perfection — and why that reframing is what earns leadership the confidence to move an agent from pilot to production.
Key Takeaways
Capability and trust are the same question in production
Splitting AI into “can it do the task” and “will we trust it” holds up in a demo and breaks the moment a customer is watching. When a wrong output can cost the account, a system that nails a demo but cannot demonstrate its reliability again and again is not yet capable in the sense that matters. Proving trustworthiness is part of the job the system has to do, not a separate approval step that comes later.
Rules-based automation dies in the long tail of special-cause work
Welocalize had already automated the common-cause majority of its project tasks with rules, but more than a million a year remained — the special-cause requirements that differ by client and by program. One customer’s single program ran to a 56-page SOP that became a 70-row spreadsheet when decomposed. Any one row looks automatable; writing and maintaining bespoke rules for all of them across hundreds of clients produces a mountain of technical debt no team can carry.
Watch people work before you automate anything
Before writing a single rule, the team ran what Sekac calls an ethnographic study — literally observing project managers do their jobs rather than interviewing them or trusting the documentation. Written SOPs rarely match what teams actually do day to day, and that gap is exactly where automation quietly fails. The observation is what surfaced the real complexity the system would have to absorb.
The shadow test turns confidence into evidence
For 30 days, the agent mirrored every human decision on live tasks without a single customer seeing its output, producing a direct, low-risk comparison against real human work. That evidence, not the team’s belief that the system was ready, is what got leadership to approve production. Plain-language instructions to agents replaced hundreds of bespoke scripts, widening who inside the delivery organization can build automation at all.
Benchmark against human error, not zero
Nobody asks whether a human process is error-free before automating it, yet everyone expects zero errors from AI — and that mismatch is the real reason trust conversations go sideways. The workable standard is not perfection but a validation step fast enough to stay a clear win over doing the task manually while still catching mistakes before a customer feels them. If validation costs more than the manual process it replaces, the automation has not saved anything.
A checking agent can get reliability exactly backwards
The team built a second agent to predict which outputs a human would override, and its confidence ran inversely to accuracy: the outputs it was surest about were the ones humans changed most. The reason is revealing — many overrides happened because the agent skipped a required step in the process, not because its output was wrong. Overrides often expose a process gap rather than an agent failure, which makes honest measurement of time saved harder than it first appears.
Related: AI Strategy | Process Automation | Episode 15: AI’s Experimentation Era Is Over, Says Davos — with Matt Sekac
Full Transcript
Jason: This is Eventual Consistency and I’m Jason, the show’s producer. Ross sat down with today’s guest so I’m here to set up the conversation. I’ll be back at the end to wrap it up. Everyone split AI into two separate questions: can the model do the task, and will anyone actually let it run? Ross’s guest today thinks that split is mostly false. His argument is that for anything headed into production, capability has to include whether the thing can prove its reliability again and again while a customer is watching. A system that nails a demo but can’t show that isn’t capable yet in the sense that actually matters. Trust isn’t a layer you bolt on afterwards, it’s part of the job that the system has to do. Matt has spent the last couple of years testing that idea for real at his company Welocalize, where a wrong output can cost you the account, so there’s just no room to assert that something works. He’s had to make it provable. Here’s Ross with Matt Sekac.
Ross Katz: Matt, welcome back to the podcast. Can you start by introducing us to the agentic AI implementation that you’ve been helping to lead at Welocalize?
Matthew Sekac: Absolutely, it’s basically around the management of the projects, right? The actual production of translated localized outputs, that does involve AI in a couple of different ways. But this agentic AI initiative is aimed at project task management. Specifically where it was too varied for rules-based systems to handle, or at least to handle at scale. What we did was to go and pretty extensively inspect what was actually going on in the population of project managers in the delivery organization. What does it actually take to take an order and get it to delivery? And we even went so far as to conduct, you might call it an ethnographic study, literally just to observe a lot of folks doing their jobs in order to see what’s really going on. And what we found is this variation across clients and across programs, and so what that means is different customers have different requirements, different customers have different expectations that we have agreed to along the way, often in the interest of wanting to provide a white glove bespoke service. But then even within customers, of which there are hundreds, right? You have different programs which are different workflows, different content types, different materials that have different requirements, that might require client-side validation, that might require an additional layer of review, whatever it might be. And those in turn had bespoke requirements. What we found was if you look at any one thing, you would say that could easily be automated with a rules-based old world before Claude agent, right? Or before LLM agent. But there are just too many to really think you could realistically put your head down and brute force your way through all of them. Not only would that take forever, but then you would have this mountain of technical debt, right? You’d have hundreds and hundreds of agents that were bespoke to this one specific requirement. So the unlock was the famous quip, the coding language of the future is English, right? If you can establish a framework where you can just give plain language instructions to agents on what to do, then, with a whole lot less technical debt and one centralized platform, you can enable a whole lot of variation, right? And you can potentially address that with automated agents. Now there’s still a whole lot of work to do to actually make that real. But if the actual cost center is coordination, right? The actual production of the outputs is just a buy-sell spread. But the actual cost center for our organization is the fixed cost of coordinating and executing and stewarding all of these projects, thousands and thousands, a huge volume of projects against the amount of revenue that we have. It’s work rules and routing and exception handling and all of that stuff. It’s, can we build agents and use their reasoning capability to absorb these things? And that does change the job for the people around it and what’s expected of them. It’s not just what the software does, but that’s the idea.
Ross Katz: Yeah, that makes sense. Just to ground it in your industry. So each of these programs is a set of localization projects to be done, which is composed of a bunch of different translations of documents of different types from a source language into a target language across all of these different parameters, with guidelines for how those translations are to be conducted and what tools are allowed to be used, what type of person needs to be recruited, etc., etc., etc. Lots of contractual complexity with each client. But you had an organization, this project management organization as I understand it, that was stood up around this that was meant to marshal any individual program or task through that complexity. So part of the groundwork that you needed to do with even just understanding the nature of the complexity that you were dealing with, so that you could consider how you would set it up not only to operate the clients and programs and tasks you have in existence today but also, to your point about technical debt, set it up so that you can onboard new clients and new programs and new tasks without having to completely rework the system or stand up a new rules engine in order to make it responsive to those contractual obligations. Am I thinking about that right or how would you update that?
Matthew Sekac: No, that’s good Ross, you understand our business pretty well, right? That’s right. Put it this way, right? It ends up that more than half, the pretty significant majority of tasks, right? Meaning for a workflow to execute a project is then decomposed into tasks, things that need doing. It was already the case prior to this initiative that the majority of tasks were automated through rules-based agents, right? But you were still left with over a million that needed doing, right? That last 30%. What had happened was I would call it common cause versus special cause, right? We’d pretty much tapped all of the common cause opportunities, right? Or even semi-common cause. This applies to these three clients or this client is big enough that we can call it common cause because of its scale. There are a couple like that where you just do whatever you need to do. But what’s left is this huge litany of itty-bitty little pieces that are these special cause requirements. Yeah, the way you described it is pretty on point, right? It’s, okay, thing comes in, we have to determine what it is, what the content type is, what resource base it goes to, which resources are skilled or qualified to handle it. That might go here, that might go there. Is there anything special about the production workflow that needs to be flagged at the beginning? Is there anything weird about the way that we price it? There’s all kinds of stuff like that. We had a client with a 56-page SOP for handling just one program and when we decomposed that from observation into steps, it was a 70-row spreadsheet, right? That’s just do a project for one program for one customer, right? You can go through each thing and be like oh you could automate that, but are you going to go through all 70 rows and write a script and agent for all 70 of them and then move onto the next and move onto the next? That was the problem and that’s where we think that the agentic capability unlocks a lot of it because a lot of it is just based on basic reasoning. That’s where it’s hard, right? Instead of having to write a rule for this and a rule for this and a rule for this, but if you could just give a heuristic and trust it to apply it and just write that in plain language then first of all it’s easier and second of all you widen the aperture of folks who can do it. It doesn’t have to go through somebody who knows how to script an agent, right? It can just be codified in plain language with some guidance by the delivery organization.
Ross Katz: Yeah, that makes a lot of sense. Can you walk us through, you mentioned the ethnographic study, I would love just hearing the story of how you went from understanding that complexity all the way up to where you are now in the production roadmap.
Matthew Sekac: The idea was that we knew that there was variation, right? The organization knew that there was a lot of variation. But it was a sort of abstract truth. It was okay, you can go and you can ask this delivery team or that delivery team or this squad or that squad, what are your protocols, what documentation, there would generally tend to be SOPs and some documentation, but then you would hear anecdotally that that is maybe not exactly what happens. That’s where the thesis was, let’s just sit and watch. Let’s document, not even ask questions maybe until later, let’s just see what actually happens and where are we manually consulting spreadsheet trackers? Where are we doing a thing that actually a rules-based agent already exists to handle? Maybe it’s not trusted. Let’s try to get the measure of what it would take to automate this, and that was before we had a lot of confidence in an agentic framework. Actually, our conclusion at the time was we’re not really sure there’s a whole lot we can do about this because, again, you could do it but the technical debt. Then, as the technology evolved and the capability of agents became demonstrable, it was well, let’s try it, right? That’s where we had to establish trust along the way, but that maybe we can do what I’ve been saying, which is harness the reasoning capability to chunk these process requirements off and then couple that with a genuine inspection of whether we need to be doing all of these things still, or is this like we’re doing this check of a check of a check because three years ago we messed this one thing up? We added a check and then there was another problem so we added a check of the check, right? If you do both of those at the same time which is not easy, then you might actually be able to get to something that starts to look transformative.
Ross Katz: What do you think has been successful about the way that you’ve implemented AI in this context, and what lessons would you take away for the next time you try to automate a workflow like this?
Matthew Sekac: In terms of how we got from ‘all right we’re going to try this’ until today, what I think was successful, and what I think gets at the theme that you’re interested in about trust, right? The way we started was this: we had a hypothesis, so what we did was we stood up a clone mirror environment of our production environment. We put the game on relatively easy to start. But we picked a client and some workflows and some tasks that we thought agents could automate. We spent about 30 days where we had the agent do the same thing that the human did. Every time a task went to a human, every time a kind of in-scope task went to a human, we stood up this shadow environment where we were having the agent do the same thing and then we compared the results. The reason for doing that was because in order to proceed further, in order to really lean in and pour resources into this, we needed a proof of concept, we needed to demonstrate that it could work. It was establishing trust with senior executives, with the leadership of the delivery teams, but also, and I think this is important, also from us, we wanted to believe that it worked, right? Because we don’t want to insist on just going full throttle until we have some demonstration that our hypothesis is correct. From the beginning we were game planning for what the path was going to look like, from this idea, from experimentation, if you want a callback. This was the experiment, right? I think this was a pretty well-crafted experiment. It was yes to establish that our hypothesis was valid, that there was at least some set of tasks that we could tackle with a LLM-driven agentic LLM agent, right? But also that we were trying to do it in a way that the data we gathered would be persuasive from the top on down. We were trying to anticipate what the gates were going to be to proceed to the next stage. It’s okay, look, we showed with this experiment it looks like this can work. That gets us through the gate into, okay, now let’s pilot it. And then it’s pick a couple of customers. You don’t want to make it like you were just really giving yourself the easiest path, right? You also, is there an opportunity to pick customers where it would have an impact? Are there customers that — the optimization formula is you’re pretty sure it’ll work and there’s a really high volume of manual tasks that you might be able to make go away, right? We were able to find a couple of those customers and we tried it on those, and it was not without hiccup.
Ross Katz: Can you give us some examples of some hiccups that you encountered?
Matthew Sekac: Yeah, one was, we have a customer that sends a lot of their requests in as emails. It sort of like bulk orders in emails and what we found was that we just needed some refinement to the parsing. It was really prompt engineering to an extent. But agents failed in a couple of cases, they got confused and just didn’t yield a result. That happened in a certain number of cases. We had some bugs in the code that fired it. Just normal stuff, right? I don’t think anything super material, nothing that would make you feel that the whole thing might not work, right? Just normal routine, oh, we didn’t account for this, we didn’t account for that, and so now we have these errors. The human element of it is something that we had to think about along the way because even now, everything that an agent does is inspected by a human, right? What’s the UI for that going to look like? How much data do we want to gather from them? The more we ask them to give input, the more burdensome that task is. So then it’s, yes, you’ve made the task go away but you’ve added another, so there’s that part of it too that we have to think through. But by and large, we’ve evaluated on the basis of, does the agent think it did its job, which is not 100%, sometimes it’s like, I don’t know what to do, it’s told sometimes to do that. I don’t think I know what to do. Sometimes it just doesn’t succeed, it doesn’t get to the end for whatever reason. And then did the human agree, right? You have, does the agent think it did its job, and then does the human agree that it did its job? And then those are your did it work criteria? We’re going to do that with two customers, and then that was successful enough, and that’s where kind of hearts and minds stuff, like the anecdotal feedback from the individuals, that was also part of the selection criteria, candidly, but the anecdotal feedback from the individuals on the ground who were using it was excited, encouraged, positive, and then it became a proper, all right, let’s ramp this all the way up if we possibly can, and that’s where we are now.
Ross Katz: Yeah, that makes a lot of sense. I’m interested in a lot of things, but how do you visualize the human in the loop portion of this evolving over time given your point about the amount of verification tasks that you assign to the project manager needs to be less than the amount of project management tasks that they needed to take ownership of earlier.
Matthew Sekac: You think about all of the different types of manual actions that you can apply this to because there’s a pretty big population of things they fall in the Mechanical Turk category, right? That where historically it’s not that it’s the most complex thing, it’s just that it’s really hard to write a rule or to write enough rules to account for it where a human can pretty easily just call it but it’s just really hard to get a computer to do it. There’s a lot of stuff in that category and that is I think the lowest hanging AI fruit, right? Is where just the basic reasoning capability unlocks a lot of that. But also one of the worst things you can do in a customer relationship is make billing a problem, right? So if some upstream automation that you put in causes you to bill the customer wrong, then it doesn’t take a lot of those to even jeopardize the whole relationship, right? So even small things like that and then it’s are you going to cause some issue in production? So to answer your question, honestly I don’t think we totally know the answer to that problem, right? I think that the way we’re approaching it is first to try to minimize the amount of time spent on the validation action so that that is at least demonstrably more efficient than having done the thing manually. For the most part that’s doable for a lot of these things, even if you rely on popular collective anecdote but ideally you’re doing it with data. And then it’s okay, so can we start to develop mechanisms to maybe have confidence intervals where we can say that this class of task based on maybe another agent is doing a quality evaluation, right? Or a likelihood of accuracy evaluation where if you can show that, okay, if you get a score higher than this the human never changes it, then you can start to unwind some of those things.
Ross Katz: Yeah, that makes a lot of sense. I think that the way a lot of companies approach the question of what are we going to automate with AI is what is the volume of work that’s being done in our business and they kind of work their way down the list but roughly speaking, I think of the cost of error versus the cost of verification and if both those things are low, if the error cost is low and the verification cost is low it’s really perfect for agentic AI because it’s going to have some error rate and it’s easy to verify so you just throw a human in the loop at it and it probably removes a lot of their work, but if it’s high cost of error, hard to verify then you’re never going to get a human out of the loop, to your point about billing just a second ago. So categorizing where these things fit in that is what I would think about.
Matthew Sekac: Yeah, I think where it’s in the more challenging end of the spectrum you can still think about, well, are there ways—if it’s hard then it’s probably complicated and if it’s complicated can it be decomposed? And so can you facilitate elements of it, right? To get you faster to the output? That’s one thing. The other thing though, Ross, is I think there’s an important thing that’s often missed. I think about self-driving cars on this a lot. It’s important to have in mind what your baseline of performance is because the thing about it is the standard should not be perfect, right? It’s if you are trying to replace a human process, then the process you’re trying to replace is already not perfect, almost axiomatically, right? Because humans aren’t perfect, right? And so I think about self-driving cars because I don’t have the data handy on this, but I’m pretty sure it’s demonstrable that in aggregate, per mile driven by a self-driving car, more control conditions, not everywhere, but you have fewer accidents, but when there are accidents, when it makes a dumb mistake or a mistake that you wouldn’t imagine a human making, that’s just felt more, right? There’s a bias towards the way that we react to and we perceive the risk of that kind of outcome even though in an aggregate you could demonstrate that it’s better, right? And the baseline is not no accidents. The baseline is how many accidents there are from human drivers, right? And what kinds of accidents there are from human drivers. And so it’s kind of the same thing here where even if it’s translation, right? Translators make mistakes too, right? Whatever it is, you have to benchmark it against a non-zero amount of error in the first place, right? And I think that’s hard for people to get used to.
Ross Katz: I think that’s right. I also think about this as when we talk about how much trust there is in a system, we’re so used to trust being interlinked with responsibility and blame. I can trust you because you’re responsible for it and you’re accountable for it and I can blame you if it doesn’t go well, whereas when you move from an individual being responsible for it to a system being responsible for it, it’s unclear and you go from errors are unacceptable but we know they’re going to happen to we know there is a proportion of errors that are going to happen and the system is just going to make an error sometimes and maybe we’ll improve over time maybe we won’t but there’s a mindshift that needs to happen there and so I’m curious what you think about that from a selling this to leadership perspective and the desire for control perspective that business leaders want to have how do you navigate that kind of dynamic?
Matthew Sekac: Yeah, that’s good, that’s interesting. No, it’s a good point, I don’t think I’d fully thought about it that way before, that what it is is that it starts to feel like you have hard-baked into your expectations that there’s going to be an issue whereas if it’s humans then you can kind of say that, when it’s computers doing it it’s like, well, the machine’s just going to mess up this often, it’s math, where with humans you can kind of try to talk yourself into the idea that they’ll keep trying, they’ll get better eventually, it’ll be perfect even though you know that that’s not true. Look, as far as winning the trust of the organization, right? And what’s the difference between trust and capability? Again, I basically think that you can’t totally disambiguate them, right? And that trustworthiness is part of the capability requirement, right? It’s not bolted on after the fact, you know what I mean? It’s part of the deliverable itself. So in the beginning, with what we did with the agentic framework, we had to convince senior leadership and the stakeholders in the organization but also ourselves, but you conceive of what convincing yourself means in such a way as you anticipate it as being persuasive enough to get you to the next stage. And that’s really what I think it is is that the capability has to be able to prove itself. And then in popular culture there is a lot of skepticism toward AI. But I think within an organization, if you’re committed to this path, if you do want to try to unlock this, you need to have human capital that is going to be persuaded by evidence and where the evidence is genuinely persuasive that should establish trust, right? It shouldn’t be a sort of emotional, abstract skepticism, it should be, okay, can you prove it works or not? And that’s a valid thing, I think, for the organization to ask, that’s a valid basis for skepticism, but once that bar is cleared, that’s a negotiation, that’s a conversation, everybody should be a team and on the same page. What does it mean? What does it look like? But we should all be moving in the same direction, right? And the trust is built together, right? With just well-conceived steps of the process where we are thoughtfully gathering data and not just saying, I don’t know, we think it should be able to do this.
Ross Katz: Yeah, I think I have a couple questions on that, but the first one that comes to mind for me, you mentioned gathering evidence and making sure that the nature of the evidence that you’re gathering is persuasive. Can you talk a little bit about how you did that concretely in this project, how did you know what evidence you wanted to gather in advance and how did you set expectations with your leadership about what the set of evidence would need to look like in order to prove that you were ready to go to the next phase of productionalization of this?
Matthew Sekac: Yeah, it’s a good question and I’ll be candid that I’m not going to sit here and pretend that we’ve solved all of it so far, we’ve definitely been thinking about it and talking about it from the start. I think our first order set of data was really about does it seem to have done the right thing, is it not messing up?
Ross Katz: An accuracy rate? You compare the accuracy rate?
Matthew Sekac: You’ve got humans validating everything, right? So it’s just did the human say it was wrong or not? Now, even that’s not imperfect, so I’ll tell you a story. One of the things we tried recently was trying to — can we put another agent on top of the agent that’s doing the work in order to try to check its work. And say, okay, these are the ones that are likely to be right, these are the ones that are likely to be adjusted by a human. We actually found in just the first try, but it was inversely correlated, meaning the more likely it thought that it was not going to be changed by a human, the more likely it was to be changed by a human. So that’s bad. It’s also flabbergasting, right? Because you’re using the same basic LLM technology to follow the instructions in the first place, so then why, when you then ask another one, how could it be so wrong? One of the things that we found though was that some of the reasons for the override of the initial agent’s output were deviations from the prescribed process. We’re not supposed to proceed with a project without this information but we override and do it anyway. That at least is what it looks like. We’re still early days on studying this, but we know that this happens, right? But even in this context where, okay, we write instructions for the agent, the agent executes them and then it gets changed, we’re counting that as a fail even though actually it did kind of do what it was supposed to. So you have to go in and inspect that too, right? But so there’s that. It gets even harder when you want to try to gauge what the impact is, right? Because depending on how robust your systems are, how reliably can you gauge what the actual time savings are, what the relative investment of energy on having done it manually versus having to validate it? I think what we’re starting with is trying to demonstrate as rigorously as possible that we can get the right outcome. And we do have pretty successful results there. And then from there I think you can start to get into, in collaboration with the business, looking at tasks that are automated versus tasks that are not. And then even if what you have to do is go watch and see what it looked like before and what it looked like now, or you can start to come up with ways of trying to estimate those things if you need to.
Ross Katz: Yeah, that makes a lot of sense. I think that the challenge as always is that — and you mentioned this several times — is that you’re going to get nowhere unless you’re actually looking at the data and looking at the outputs that the agents are producing because even if you have a human in the loop coming to conclusions over and over again, you can trust that the human is right but the human is making decisions not just based on what’s strictly written there, the context that’s on the page that is available to the agent, but based on context that is a superset of that. And so you have this additional tacit knowledge that you were describing that comes into the picture about how you do business, what this customer actually cares about versus what’s written into the contract, habits that have been developed over time about how you work together with your customers that have never been documented anywhere but have not made it into the instructions and so there’s all of those edge cases that need to be sussed out by being grounded in the data directly but then also over time you’re also trying to create firmer and firmer methods of condensing that information into something actionable so that you don’t have to be looking at that data to that degree of detail in order to understand what’s going on in the system. So I guess the question that comes to mind for me is, do you have a feel for what the timeline looks like to — I don’t know what we would call this — but take the training wheels off so that you’re not having to as tightly manage or tightly observe the agentic system for these edge cases that arise?
Matthew Sekac: I think if you ask my bosses it’s quite fast, we have to figure a lot of this stuff out. I think one thing that’s easy to forget is just how quickly the technology has been developing where we’re having a conversation about all of this stuff is established and longstanding, it’s just part of the status quo. But in fact we’ve had Claude for less than six months, we’ve been working with the agentic AI for, okay, 18 months and 18 months seems like a really long time in this world, right? But that’s conceptualize, build prototype, get to pilot, do pilot, right? It’s also a process engineering initiative, right? I think we talked about this a little on the last podcast but the approach shouldn’t be to just take everything you’ve got a human doing and one by one automating it, right? It’s as part of this process you do need to re-evaluate. I think you had the metaphor with the nail, right? You have to re-evaluate whether those things are actually necessary to do. Some of them might have been unnecessary to do even without this AI initiative that you’re in the middle of, right? As far as timing, it depends on how you would define the endpoint I guess. We’ve gone from a couple of customers to dozens of customers where we’ve pretty rapidly scaled the number of projects that this touches well into the thousands, right? But because of the original problem it still needs to be that folks in the organization have the capability and the knowledge and the training I guess, they at least need to be oriented in terms of how to configure these agent swarms in order to tackle more and more chunks of customers in order to automate as much as possible. And then you’re also looking at different types of capabilities, right? There’s pre-production activities which tend to be on the easier side, then there’s production activities, what resource does it route to, in what way does it route to them, those sorts of things, and then there’s post-production activities, and so which of those do you tackle first, right? So it’s not just which customers are you tackling but which pieces of their workflows are you tackling? So I think we’re still well into next year bustling on this at minimum.
Ross Katz: Yeah. Well it also sounds like in addition to the specific project management initiative you have going on with agentic AI, there’s also using this as an example to build up agentic engineering practices internally, agentic platform capabilities that allow you to apply this across domains, not just specifically to project management. This is more of an engineering team problem but it’s how do we take what we’re doing here and make it so that it’s repeatable in other domains of the business? Am I thinking about that right?
Matthew Sekac: Yeah, I think so. Ultimately what you’re looking at are repetitive manual tasks, right? Which even—okay so you compare current state to 10 years ago, you’d say they look a lot more automated, largely driven by a material investment in rules-based agents, where there are common cause activities that can be automated away, right? But that’s true in other areas, not just in production and delivery for our organization. Finance is an example where generating POs and creating new accounts in our ERP, or just billing and collections, things like that, right? Those tend to involve a lot of manual activity too, so you can apply the same lens. Another thing is one of the things this new AI technology does do is it democratizes—it really does democratize what a motivated curious individual in the business can do, right? It’s curiosity, data literacy and a rough technical aptitude. You no longer have as hard a blocker that just comes from was never trained in SQL or Python or JavaScript or whatever it might be, right? You just need to have an idea for roughly what should be possible and how to—and just the logical thinking, right? And then access, and you can at least mock stuff up, right? That gives us a head start towards production ready value added tools.
Ross Katz: Yeah. No, what you’re saying is really interesting. It also highlights for me the headlines that you see are the gap between demo and production. But I think that’s framed as AI doesn’t do what it says it’s doing, but I actually think there’s two—there’s the numerator and the denominator of that equation: there’s the number of demos that get produced in order to demonstrate value for different use cases and then there’s the quantity of production of use cases that ever make it into production. I think you would expect from what you just said that if more people, if the entrepreneurial scrappy and curious business user has the ability to create more and more of these demos to demonstrate value, then the fact of the matter is even if that person has a 10% hit rate but they still produce 10 demos, if one of the things that they create makes it into production then you as a business are in a much better place than you were otherwise. What do you think about that?
Matthew Sekac: Well I think one thing you have to be careful about in that framing is that can go wrong in that direction too, right? That can get into—it’s not exactly AI slop territory, but I don’t know if you’ve observed this Ross, but one of the things that does end up being a little bit of a challenge with the ability of AI to generate nice-looking pretty high-quality content so quickly is you end up with a really lot of nice-looking high-quality or at least seemingly high-quality content, but which might have little errors—it’s hard for the person who produced it to really have spent the time to consume it and then they’re distributing it and then they have to consume it and you end up with just a lot of content to consume, right? But I think you’re right. I think what you’re hoping is that such a user would first self-screen their brainstorms. But I think the more important thing is when we say ‘demo,’ what we mean is something closer to ‘done’ than what we would have meant by ‘demo’ before, right? Demo before might have been a PowerPoint presentation that I visually mocked up or I’m using one of the tools that are out there that let me hypothetically stage a click-through, right? That takes a long time to prepare and it’s not actually coded in any way that is connected back to whatever underlying database or system it needs to work off of. Whereas you can mock up the thing—it’s almost done, right? And then all that’s left is—hell you might be able to deploy it. Now we have examples of that, automating certain processes where all we have to do is enable a connection or enable the ability to make a write through the API to Workday on a certain fixed set of records and then now this thing that used to take a long time—and this is a citizen developer, right? But here’s an automation that you put the stuff in and then it goes and it creates these artifacts that we need in Workday and it does that quickly, right? And it’s done, right? It’s not a mock-up or a demo. You demo it and then what’s left is permissioning it or provisioning it or making sure that the API is enabled, right? There’s stuff like that. I think for a product manager, this full stack PM concept, right? They can almost prototype—they can pretty much prototype the story, right? And then the engineer still needs to validate that it complies with the rest of the code, that it fits into the broader tapestry of our codebase and stuff like that, but you can—your demo is a thing that works. That I think is—it’s not so much the volume of things it’s the quality and realness of them that it either makes them more persuasive it makes them more likely to get on a roadmap but then it also is more of a head start.
Ross Katz: Yeah that’s really interesting. It also highlights for me the nature of the business that you’re operating in and the extent to which your systems are architected to give the people doing the work visibility into the entire workflow and the opportunity to experiment in this way is also another consideration that companies have to have when they think about the way they design their roles: is there meant to be the flexibility to experiment inside of this role or is the role the recipient of whatever the AI workflow is. But are there aspects to this that you think are the most important things that you’re trying to learn over the next year as you think about improving the efficiency of the work you do at Welocalize or improving the quality of the implementations that you do from an AI perspective?
Matthew Sekac: I think when we get into really trying to do this, it really is about trying to think in advance about how you’re going to know if it worked. We’ve really tried to do that but then you hit unexpected snags and a lot of it is details, right? That’s the thing is you need to be steeped in the details. It’s hard to do this kind of thing in the abstract, right? I think that’s across the board when it comes to anything you’re trying to develop for an organization: the more it’s done in a vacuum the more likely it is to not be accepted or used or adopted because there’s some misalignment in understanding, right? You still have to know the business you are serving and the better you know it the more you can anticipate objections or sources of mistrust, right? But really, what’s the best way—you’re not just trying to build a thing, you’re trying to have an impact on the business with the thing. You really want to try to think about how the thing you’re setting out to build is actually going to have a demonstrable impact and if that means you think you can anticipate objections you’re going to have to overcome or resistance you’re going to have to overcome which are probably often valid. You satisfy yourself first and then the leadership and then the line level, right? But you want to think all the way through that because if you don’t have that you’re going to find yourself in a tough spot where, okay, you can have strategic insistence from the top on down but then you better be right, right? I think we’ve done a reasonably good job about that but it is hard to measure the impact a lot of these things, particularly depending on how your pre-existing data structures enable you to set a baseline. We don’t want to do that without data and it can require some creativity to come up with how you’re going to structure your experiments or gather information along the way and the more that in the beginning I don’t have this data that I know I’m going to need, do I have an opportunity as part of this to start gathering it? Because I’m not going to make this decision in three months, it’s going to be six months but six months will be enough data so can I just start gathering it now knowing that it’s not going to be useful for half a year but I’m going to want it. Thinking about that from the beginning as much as you can. You start at the beginning and the end with things like this.
Ross Katz: No, the term that gets thrown around is evals-based development, and understanding how you’re going to evaluate yourself and how you’re going to capture the information to evaluate yourself is a critical precondition for building in the first place because if you can’t evaluate then you can’t gain the organizational momentum that you need to see something all the way through.
Matthew Sekac: You also can’t hold the organization accountable, because that’s part of it too. It’s not just all you evaluating yourself is in the end it’s you’re serving the organization and the organization needs to see value and it’s not just about winning them over to something that you think is great, they need to be bought into it and they need to be signed up and on board with it because ultimately whatever P&L or business impacts that you’re having that they’re the ones who have to manifest them.
Ross Katz: Well Matt it’s been great to have you on. Thank you for talking about this. Would love to have you on again in six or 12 months to talk about how things are progressing because the more I talk about initiatives like this the more it starts to feel like these are multi-year long initiatives that have to be ushered all the way through and the conversation about AI automating all work out of existence within six months is overblown and difficult to square with the reality of trying to put AI into practice inside an organization like Welocalize.
Matthew Sekac: It’s still a business problem. It’s a business process problem, but with just exciting new capabilities.
Ross Katz: Totally. Thank you, Matt.
Jason: That was Ross with Matt Sekac. The point that really stuck with me most is that the baseline was never zero errors, it’s whatever error rate the humans were already running at and pretending otherwise is what gets teams stuck comparing AI to a standard nothing has ever met. What’s clear now is that proving a system works: the shadow runs, the human agreement checks, the boring measurement isn’t the overhead before the real engineering starts, it is the engineering. You can find Matt at Welocalize and if you’re the one being asked to prove your own system holds up before anyone will trust it with production, this conversation was for you. If you know a data leader stuck arguing capability and trust as if they’re two different problems, send them this episode. And if you think we got it wrong, tell us. You can find us at www.corrdyn.com. See you on the next one.
