Listen on
The declaration from Davos is stark: the AI experimentation era is over. After trillions in investment, nearly two-thirds of companies have not scaled AI beyond pilots, and boards now demand measurable business outcomes. This shift forces data leaders and executives to move past exploratory projects and deliver tangible ROI, a challenge that requires more than just adopting new technology—it means re-evaluating core business processes and data foundations.
In this episode of Eventual Consistency, we discuss what this mandate truly means for organizations building and using data systems. Matt Sekac, who has led data and analytics at Welocalize through the entire AI hype cycle in the rapidly evolving localization industry, offers a grounded perspective. He explains how his company integrates AI deeply into service delivery and product development, emphasizing how foundational data infrastructure and a clear operational model are critical to moving AI from concept to production-level value.
Ross and Matt explore the nuances of disciplined experimentation, the necessity of process redesign for true AI impact, and the difference between strategic leadership and micromanagement in AI deployment. They also examine the role of AI Centers of Excellence and why AI’s value extends far beyond simple cost reduction, encompassing developer productivity, enhanced product offerings, and competitive advantage. This conversation is for leaders seeking to transform AI investment into sustained business growth.
Key Takeaways
AI’s ‘Experimentation Era’ Demands Discipline, Not a Complete Halt
The message from Davos isn’t to stop exploring AI, but to transition from activity-for-activity’s-sake to disciplined experimentation. Leaders must set a higher bar, moving from ‘did we try something interesting’ to ‘did we learn something that informs a decision or a real investment.’ This requires clear value hypotheses and a defined path to production from the outset.
Achieving AI ROI Requires Process Redesign, Not Just Tool Insertion
Simply dropping AI into existing human-centric processes will not yield significant returns. True mechanization of knowledge work means rethinking inputs and outputs from a blank sheet of paper. Organizations must identify needless redundancies and redesign workflows around AI’s capabilities, ensuring systems are built for automation from the ground up.
Leadership Drives AI Value Through Strategic Insistence, Not Dictation
Top-down imperatives are crucial for transformational change. However, CEOs must provide ‘strategic insistence’ by pointing the organization toward high-value areas for AI application and demanding value hypotheses. Micromanaging specific technical implementations or issuing a blanket ‘use more AI’ mandate often backfires, undermining buy-in and hindering tailored solutions.
AI’s Value Extends Beyond Simple Cost Reduction
ROI from AI manifests in diverse ways, including increased developer productivity (e.g., 30% faster code shipping with fewer bugs), enhanced product offerings, and accelerated speed to market for content (e.g., 50% faster for one client). Measuring this broader value often requires redefining quality benchmarks and assessing competitive positioning, not just direct cost savings.
Related: CorrDyn assists clients with AI strategy that delivers measurable business value. We also focus on building robust data engineering foundations that make AI systems reliable and effective. For deeper insights into leveraging AI, see our article on 7 Principles for LLM Business Cases.
Full Transcript
Jason: Welcome to Eventual Consistency. I’m Ross, my guest is Matt Sekac from Welocalize. Every week there’s another announcement about AI changing everything, another framework that’s going to solve all your problems, another hot take about how you’re doing data wrong. We’re not here for that. We’re here to talk about what’s actually happening, honestly, the way you’d talk about with your team. Today on the podcast, Davos decided that the AI experimentation era is over. CEOs are demanding ROI. So let’s talk about what that actually means from people building or using data systems.
Ross Katz: All right, the big story from Davos this year, business leaders have officially declared the AI experimentation era over. The message from the World Economic Forum was pretty blunt. After 1.5 trillion in AI investments last year, McKinsey found nearly two-thirds of companies have not scaled AI beyond pilots. Boards are done accepting, quote, “AI activity” as evidence of progress. They want measurable business outcomes. Siemens chairman, Jim Hagemann Snabe, told executives that CEOs need to be, quote, “dictators” on AI deployment. That’s a strong word choice. According to Celonis, companies with AI Centers of Excellence see 8x higher returns than those with distributed approaches. Celonis’s CEO says you need three things for ROI in AI: strong leadership, that Center of Excellence structure, and live data connected to AI platforms. OpenAI and Anthropic are pivoting messaging toward enterprise ROI, with 40 to 80% of their revenue now coming from corporate customers. The success stories being cited at Davos include Siemens saving 30 to 100k euros per factory station with EthonAI, Nestle Purina getting full payback from Boston Dynamics Spot robots within a year. But here’s the thing. Those companies had decades of operational excellence and investment before they touched AI. The AI they were referring to isn’t even large language models, for the most part. Matt, you’ve been leading data and analytics at Welocalize for six years now, including through the entire AI hype cycle. What was your first reaction when you heard the experimentation era is over?
Matt Sekac: Look, I’m being glib. I do understand the idea here and I think if you look into the specifics of what a lot of the CEOs were saying, a lot of them messaging, I don’t know if it’s quite that severe. But I do understand what they’re trying to say. I don’t think that the experimentation era can be over if we’re starting from the premise that we haven’t solved the problem of how to unlock all of the value potential of this new AI technology. If we haven’t solved the problem, you can’t just wave your hand and make the need for experimentation go away. What is over, or I guess what it sounds like folks are trying to put an end to, is the idea that activity counts as progress in and of itself, that you can do a pilot just for a pilot’s sake, that you can not have a clear value hypothesis or an owner or a path to production and say that that’s success. From my seat, experimentation doesn’t go away, it can’t go away, but it needs to get more disciplined. The bar changes from did we try something interesting to did we learn something that actually informs a decision or a real investment.
Ross Katz: Just building on top of that. For a long time, turning back the clock to 2022, we didn’t really know what these tools were capable of. Since 2022, the capabilities of the tools have been evolving so quickly that some degree of experimentation was necessary in order to even understand what capabilities were present.
Matt Sekac: It’s easy to forget that we’re only three years into this technology even being widely available and existing. You’re absolutely right in terms of how quickly things evolve. We were recently having a conversation about how you almost can’t even have a 12-month roadmap anymore because I mean you can, but then what you end up doing in six months, what’s available, what the priorities are, the market dynamics change that you’re facing as an organization. Yeah, I think that’s right.
Ross Katz: Yeah, I think that’s especially true of your business. Before we get too much further, can you give an overview of Welocalize’s business and how Welocalize is using AI today and then maybe a little bit about your role too?
Matt Sekac: Welocalize provides translation, localization, and AI training data services to corporations all over the world, including a lot of the world’s top tech companies. At Welocalize, AI isn’t, I would really say it’s not something we’re just experimenting with on the side. It is increasingly embedded in how we deliver services, how we build products, how we think about building new products, and how we run the business. We use it all kinds of ways. One is directly in our core translation and localization workflows, things like machine translation. AI actually predates ChatGPT and generative AI. But quality estimation tools and now using LLMs and generative AI to implementing systems like that that sit alongside older technology and humans to improve speed, quality, and consistency of products, to deliver more value to customers. Then internally, when it comes to platform and product, in how we actually deliver services and how we deploy our resources for customers, we’re building AI-enabled systems that help to orchestrate work at scale using agentic technology, making decisions about what gets done by machines, what gets done by humans, how that all fits together, what a good output even looks like, and how you’re dealing with customer priorities. Then internal enablement. Helping our product teams write better stories and better epics. How we ship code faster. We’ve seen real productivity gains using generative AI in that area. How we make decisions more effectively. My role sits across all that. As you said, I’m in charge of data and analytics for Welocalize. That leaves me with maybe a broader remit than is obvious within my organization. But I end up less focused on any one model and more focused on the ecosystem. The data foundations that underlie all of those applications, the operating model, and how we measure impact on the operating model, what the guardrails are, and really connecting all of those things I’m talking about back to business reality. In a way that lets AI actually create value in production, and this is going to be one of the things we end up talking a lot about, rather than just looking nice in demos.
Ross Katz: Yeah, that makes a lot of sense. Just for people who are not familiar with the localization industry, some examples that come to mind for me are you’re a multinational corporation rolling out marketing campaigns across different languages and you want the outputs to be culturally sensitive and also relevant and also sound right in whatever language they’re being in. Or maybe you’re a pharmaceutical company and you need to do regulatory filings in a variety of different places. There are innumerable situations in which you need the outputs to not seem like you just threw it into ChatGPT or into Google Translate and got the output because you can’t stand errors. Am I thinking about that right?
Matt Sekac: It almost doesn’t occur to you that it exists until you find out that it does and then it’s like of course that exists. You think about how much of what companies are these days is words on a screen in the end. Yeah, absolutely. One of the most intuitive ones is a marketing campaign, where if you are a global company and you’re doing business in 100 countries, then you want your message to get out in those languages. You want it to be culturally appropriate. You don’t want it to just be a straight translation. You want to make sure, obviously the classic example is the Chevy Nova, doesn’t go. But there’s more subtle stuff like that that can come up, across cultures, across languages. Just even your user interface. If you’re an architect who uses AutoCAD, all of those words on the screen are in English. You would expect if you’re an architect using that tool in Brazil that they would all be in Portuguese. Someone had to do that and it’s important that, whether it is UI strings or informed consent forms for a clinical trial that we’re doing in Argentina and we’re doing in India, it’s really important that that material be accessible, understandable, and get the same message across to the target audience, and that ends up being hundreds of languages in a lot of cases.
Ross Katz: That makes a lot of sense. Turning back to experimentation in AI, one of my favorite books that I’ve read recently is this book called The Origins of Efficiency by Brian Potter. It’s this book that traces the history of mechanization, primarily in manufacturing processes. One of the ways I really like to think about AI and LLMs in particular is that we’re in this moment where this is the beginnings of mechanization of knowledge work. What I love about what Potter talks about in The Origins of Efficiency is this idea that you don’t mechanize a process by basically sticking a machine in and asking it to do the same thing that the human was doing previously. The way that we manufacture nails today is very different than a blacksmith hammering the head and then attaching it to a nail. It looks different. You have to redesign the process and reimagining that process is not something I would imagine without experimentation. You’re in this industry where translation has been under disruption from AI for a good little period of time here. But then also there’s this opportunity to take an industry where you are using humans and their expertise selectively to make sure that the quality remains high, but that requires mass project management across varieties of different company inputs and outputs that need to be produced. I’m interested in hearing from you number one, how does that redesign a process resonate with you in terms of mechanization and then how do you think that influences the type of experimentation that you’re doing versus the type of things that we have to move beyond experimentation?
Matt Sekac: I think that’s a great analogy, a great comparison. It’s absolutely true. What that also leads to is if you can’t make your process for manufacturing nails more efficient with the machines, that doesn’t mean the machines aren’t good or don’t have value. It really does compare nicely to what I think is going on here where I think a lot of problems — the theme of a lot of this is alright, we’ve been pouring a ton of money into this stuff. Okay, come on, when do we get the money back? We need that R. There’s been a lot of I, where’s the R? I think in a lot of cases they’re not AI technology problems. They might be orchestration technology problems. They might be data technology problems. But a lot of times they are process problems. I actually think that with LLMs, it probably is closer to possible that you can just ask an LLM to improvise the same thing you’ve got a human doing, but that’s also wrong and not how you’re going to get ROI. It’s maybe not impossible, but it’s not the best way to do it. One of the main initiatives at Welocalize that we started as an experiment last year has evolved now into a more focused initiative with P&L impact expected against it. It is using agentic AI to automate certain classes of task that are just difficult to automate with traditional rules-based agents. There’s a bunch of different reasons for that, but a lot of it comes down to the fact that you have people spending time doing things that should be automatable but are so varied across whether it’s across customers or across workflows or across programs or across service profiles or whatever that you could maybe do it but you would end up with a ton of technical debt and it would take a ton of time. But where there’s a sort of flexibility that you get from the reasoning capability of LLMs that gave us this hypothesis that we could use agents to actually go in there and make them configurable not necessarily by a trained developer. The famous quote, the programming language of the future is English. There’s still a lot of work to do to set up the orchestration platform for that. There’s a lot of work to do to make sure that we have data that helps us validate that things are working, that helps us make sure that we’re not causing problems, and that helps us actually measure and assess whether it’s having an efficiency impact. But then the other thing you have to do is you have to really look at your processes and ask whether that is the thing that you want to try to automate or whether there are actually needless redundancies in there. Are there checks or extra steps or offline trackers or things that maybe made sense three or four or five years ago? The fact is that, and I think this is probably true of a lot of companies, your processes weren’t designed with these capabilities in mind. I think it’s an opportunity though because when you get into it, it also requires real buy-in. It requires everybody to be aligned and moving in the same direction and interested in driving, cooperating, and asking questions and challenging what the status quo is. Then thinking about okay, less about how are we going to automate this sequence of steps and more almost starting from blank sheet of paper and saying I have these inputs I need to get to these outputs. What’s the best way to do that? Obviously you’re going to end up using a lot of the same steps because there are going to be things that have to happen, but really just looking at it that way, that’s a good mental model or frame for cutting some of the fat if that makes sense.
Ross Katz: Yeah, it makes a lot of sense. A couple things that I heard from you there, number one is that you’ve got people processes that have built up over years to account and compensate for the strengths and weaknesses of humans and use humans, which is the ultimate general-purpose technology, to accomplish a particular process. But ultimately what you’ve done is you have to start with boiling it down to the inputs and the outputs and then knowing now that we’ve done this exploratory experimentation with these models to know what their capabilities are, we have to look at that blank sheet of paper and ask ourselves what would this process look like if we wanted to automate it from beginning to end. That has implications for not just the architecture of what kinds of agents you’re using, which tasks you’re assigning them, how you’re orchestrating them, how they’re interfacing with each other, but also where is the human essential? The other thing I heard was, how do we gain the monitoring and the observability of the system so that we understand whether we’re actually getting value from this new approach that we’re doing but also we can improve over time. We can diagnose the problems that the new system has and we have a flywheel that’s going to get this process more and more effective over time. That’s one of the places where it’s really hard. Please.
Matt Sekac: It’s also not just about eliminating these tasks or whatever. You could look at some of the things that are going on and you see all these clicks that are happening. It’s I gotta go here, I gotta go here, I gotta go there. One question is is that how I want to spend my human capital? Is that where they add the most value? The other thing with respect to that last bit about trying to make sure that there are structures and methods in place to validate that the agents are being successful, it helps to build confidence in the user and in the individual because these workflows that we’re contemplating do retain human in the loop to validate because even small mistakes — it’s maybe not the end of the world but it can jeopardize a relationship with a customer, and it can undermine confidence not only from the customer but also within the organization that you’re trying to make better. We’ll get to the dictators comment I guess, but you do really need persuasion and buy-in I think for these things to be really successful.
Ross Katz: Siemens chairman says that CEOs need to be dictators on AI deployment. What do you think about that?
Matt Sekac: Well, look, the first thing I thought I guess was maybe a little pedantic because it was wait, aren’t CEOs always already dictators? They don’t have a legislature, judiciary in most corporate org charts. But here’s the question: what do they dictate? How opinionated are they? How far down do they exert influence? Look, I’ll say this: I do think there’s some validity to this in the sense that I maybe would not choose that language. But what I think our experience would validate is that you do need an imperative from leadership, a top-down imperative to really drive transformational change. I just say that I don’t think that that’s particularized to AI in this new technology. I just think that that’s always true. It’s always true that if you really want transformational change, there does need to be consensus and support. But consistent messaging in an imperative from above for that to really happen. I’ll give you — when I saw this, I went down a rabbit hole. I was reading about fax machines. Depending on where you are, your doctor’s office might still use fax machines. Email has been Pareto superior to fax machines for at least 20, probably more like 30 years. There are examples in I think Japan and Canada in particular where major elements of their healthcare system were still using fax machines as the normal course of business. What it took to make it go away is somebody saying you’re going to make this go away, that’s it. Somebody giving a direct order, being a dictator if you like, and just saying we’re just not going to use fax machines anymore. I don’t care why you think we should. The rabbit hole is, well, why do they still want to use fax machines? The answer was basically if it ain’t broke don’t fix it. I think that age-old saying is actually maybe broken. It actually might be broke because that mentality doesn’t play for what I think enterprises want to accomplish with this technology. Things that don’t feel broken, you don’t want to fix. Humans — it’s a human thing. There’s a resistance to change. I do think overcoming that does require a top-down imperative, but as you pointed out that’s strong language. Top-down imperatives matter but a fully cooperative posture without any buy-in, there’s a spectrum between coercion and persuasion, and abandoning persuasion I just think that can backfire. You want strategic insistence, not just base compliance. Because in the end you’re going to have to delegate something. You could be a dictator or wherever you want to say, but in most organizations the chief executive is not going to be able to make every single decision for every single employee. There are going to be strategic decisions that need to be made along the way and everybody being on the same page and bought in is part of that too.
Ross Katz: Yeah, that term strategic insistence really resonates with me. What we do is we go into organizations and we assess and develop and deliver and implement data infrastructure and AI-driven systems. One of the main differentiators between a project that we expect to be successful and a project that we don’t expect to be successful is are we connected to the leadership of the organization who will ultimately determine what the value of the system being implemented or the process being implemented is? I love that strategic insistence because CEOs know in some ways better than anyone else in the organization what are the sources of value in the organization, what are the sources of cost in the organization and so they can point the organization in the direction of if we focus our attention at trying to mechanize, trying to bring these AI capabilities into our processes more effectively, we’re going to be able to see the value. But that’s a strategic assessment and ultimately that’s the place where the CEO can play. But I think there’s two ways this can go. It can be the CEO dictating across the board just use more AI, and obviously if all you’re doing is assessing the capabilities of it and seeing where it might be useful, I think that’s possibly good. But if you’re really expecting to get returns and not have so many experiments fail, then being more targeted, being more strategic about that posture is going to be useful. The other way is micromanaging what the business does and saying you business unit, you need to implement AI to automate this particular process and it needs to be agentic. It could be actually that the efficiencies that business is going to get are going to come from better integration between their existing software systems, a supervised learning model that makes a decision that humans are making or that is being made inappropriately right now, and putting LLMs in the mix there is naturally going to fail. I just think that obviously CEOs care about marshalling the resources, the expertise of the organization to accomplish these really important strategic goals, but obviously when you talk about headlines from Davos and stuff like that, some stuff can get lost in the mix.
Matt Sekac: That’s what’s going to happen because actually what you just described is going from a just use more AI to okay but now we need a value hypothesis when we’re going to make a new investment and focusing our attention and energy in those directions insisting on that. I think it’s natural though. I’d say that’s how we’ve always looked at it, but it’s not going to happen by magic.
Ross Katz: I also just think that it’s important that every organization and every leader within the organization knows where you are on the explore versus exploit spectrum with regards to this technology, because there are emerging use cases that should still be explored and if you don’t explore them then there’s going to be nothing to exploit later. But what we’re hearing from largely CEOs and chairmen of big enterprise is that the time for exploration has come to a close and the time for exploitation is here. I think that typically these organizations have already invested a lot of resources in understanding their process, in mechanizing their processes. So it makes sense to me that they would be closer to exploit than organizations in the middle market or organizations more toward the startup phase where the capabilities of these tools, if you design around them rather than trying to integrate them into your existing business processes, there might be something transformational that can happen. Obviously, if there are more opportunities to find something transformational you should be doing more exploring. If there’s more opportunity to get value from further automating something you know already works then that’s where you should definitely be on the exploit side of the equation. I think that sometimes gets lost in messaging like this too.
Matt Sekac: I agree with that.
Ross Katz: Cool. We mentioned earlier that there’s an 8x ROI difference, I think this was from Celonis’s CEO, that there’s an 8x ROI difference from having an AI Center of Excellence rather than having implementation be distributed. What do you think about that idea?
Matt Sekac: Honestly I think that actually goes back to what we were talking about before. It’s in the same vein. An AI Center of Excellence isn’t magic. My guess is okay, that was based on a study. If all the organizations that didn’t have one just stood up a committee to meet every couple of weeks and called it an AI Center of Excellence that would not itself have really had much of an impact on the outcomes. In some ways having a really effective Center of Excellence is maybe as much result as cause of it because it’s connected back to the operational maturity we were discussing before and companies like Siemens and Nestle have laid a lot of the foundations already. They’ve built operations that are rigorous and matured and built processes that — we were talking about the analogy with the nail and manufacturing that versus a human process and in how they looked for us as well. If you’ve done that already you obviously have a head start. We are only a few years into this. It’s actually something we’ve talked about internally here: to what extent would we expect it to really have an impact? I think it’s really looking at what are the things that your Center of Excellence would be meant to actually deliver and then solutioning it from there. Does it make sense to have a standing committee or whatever in a meeting or does that just start to feel like bureaucracy? I think really what you’re trying to do is develop a muscle of repeated problem solving and then you are trying also to make sure that there’s a concentration of influence to help steer and evaluate. I don’t know that a formalized Center of Excellence is the only way to accomplish that, but I do hear the argument that says the more distributed and diffuse it is maybe it might be challenging. Some of it might depend on how big your organization is. The only other thing I think I would want to call out there is I do think there is a risk to over-centralizing as well and suppressing broader innovation. While we have thought about this, the truth is that one of the things we’re really thinking about lately is what can we do to actually unlock and harness — because one of the other things that was talked about is giving everybody ChatGPT is a failed experiment and there’s that sort of perception out there. I don’t really think that that’s true. I think in order for that to be true, it means that your value hypothesis for your experiment was probably off. I don’t think okay everybody here’s a ChatGPT Pro license — or I think what more organizations are doing is standing up their own walled-off instance of GPT-4 or whatever it is, but okay and then now I just expect my company to become 30% more efficient in a year or two. That’s not going to happen. But what it does accomplish is it’s foundational. I think is our point of view. It gets people comfortable with it, not afraid of it, it develops literacy, and then adoption. That’s foundational to unlocking some of these things. It does leave you with some weird counterfactuals and a lot of your ROI is going to be on a counterfactual basis. If it’s sales enablement or customer engagement, we make better PowerPoints, we get them out faster, our RFP responses are cleaner, whatever. Does that mean we wouldn’t have won that account or some of it might just be table stakes in terms of keeping up with competition? But once you’ve laid the groundwork of the literacy and adoption and engagement with it, you do I think have potential to unlock latent innovation and insight capability that exists throughout the whole organization. Our quality team does not contain developer. Okay, but first using ChatGPT and then hearing about tools like Replit and v0, these vibe coding tools, using ChatGPT to figure out how to learn these, to figure out how best to use these tools which then do code, but going from there to actually coming up with some creative solutions that we can deploy in production to some weird annoying tricky problems that come up in the course of localization. Things like it turns out that a user interface string in some cases has a rule that it can’t exceed 150 characters. But if it’s 144 in English and you translate it into Spanish then it might break the limit, and that throws errors, it pisses the customer off, we have to do rework or we have to go think about it. That’s just one tiny little example of the kind of thing that comes up, but where the quality team has managed to reduce rework, has managed to make us look better to customers, and has made it so that we are delivering a higher quality product to our customers. Done that on their own. They get some support in terms of deploying it and making sure there’s security and stuff like that, but going from just building it themselves — that’s the kind of thing that I think we want to see more of, not less. Has to be controlled obviously, there are other concerns, but you get it.
Ross Katz: Yeah, I think that makes a lot of sense and I love that anecdote. My outlook on mandates for org chart changes as solutions to problems, I’m always skeptical of org chart as a solution to any problem because organizations can have very different structures and still be successful in those structures. It’s really how does the org chart intersect with your business model, how does it intersect with the variety of business lines that you have, how does it intersect with the human talent that you have inside your organization that can manage and oversee different sections of the business. This Centers of Excellence model, it’s great when you’re more on the exploit side and you want reusable components that you can use across the organization and you know exactly what’s happening across the organization so the inter-institutional knowledge of the processes is already centralized. You’ve got the CEO mandate so they can open all the doors, they’ve got all of the API keys and all of the database credentials that they need to pipe the things together and they’ve got the sufficient resources and the freedom to experiment. But if you’re not in a situation where all of those things are available, for example, I doubt that someone in the executive team at Welocalize would have been able to say that the quality team was going to have those particular issues when interfacing with customers, nor that if they gave them AI tools they’d be able to bootstrap themselves, resolve the problem themselves, and then once you have that in place then you can start centralizing those sorts of things. You make a really great point about the balance between these things and how you create an environment where that innovation is encouraged but also make sure that you’re standardizing and that you’re exploiting as well as you can. It feels more like a two-step and this Center of Excellence red herring feels more like something that’s great when you have an enterprise-scale manufacturing organization where every factory is doing things the exact same way or where that knowledge is really well-centralized already.
Matt Sekac: Yeah, I think that’s right. It’s self-reinforcing. You see examples like that and that gives other folks ideas and then you can end up with more centralized initiatives, whether it’s blowing something like that up or taking that concept and maybe it could be applied elsewhere. I do think that some sort of centralized leadership and direction and trying to focus on looking for focused opportunities with really plausible value hypotheses, and investing there and having centralized resources to invest there, that is part of it. But stuff can bubble up that can really help drive that.
Ross Katz: Yeah. I think your status as an emerging enterprise or a mid-market organization depending on your definition puts you in a place where both are imperative and that’s a challenging area to navigate. We’re almost out of time. Is there anything on the list that you want to make sure that we get to before I let you go?
Matt Sekac: I think in this whole conversation about ROI, one of the things that I think is important is trying to develop a mature idea for what ROI means. One of the topics that we’ve been talking about is foundations versus — what things are foundational? That could be your operations process, it could be your data. Do you have the right data to measure things? But then also how do you think about ROI because it’s not just mass layoffs and cost reduction. I do think that’s the tone that people get when they hear talk about ROI, but I could tell you in our organization that’s nothing like the only one or even the main one that we’ve realized so far. Dev productivity, we saw a 30% increase in code shipped using some of these LLM-based tools that you can plug into, again with product helping with crafting better stories and epics more quickly but then also in drafting the code. That was an example of where it did take some pressure from leadership because with the development team saying y’all are going to try this and then seeing that it worked, we’ve seen a 30% increase in productivity or in output with a considerable reduction in errors. So you’re shipping more code to the organization faster and with fewer bugs being spotted in QA. Our organization, Welocalize, is really focused on AI and technology generally because as you could probably imagine large language models are disruptive to a language industry and it creates a turbulent highly competitive situation where we are investing a lot in technology to develop new products. That extra productivity helps us keep up with the rapidly escalating appetite from the business to invest in what developers do. It didn’t mean that we fired 30% of our developers. Another angle is your offering. One of our key AI applications is a product that we’re rolling out called Opal Enable. You were asking at the top about what localization looks like and talking about all this volume of content that needs to get shipped and translated into all these languages and distributed through all these channels. The way that the content generation and distribution lifecycle looks is a big part of the puzzle that we have to solve because that’s evolving too. But when we look at okay where’s a use case for AI? I think the first thing a lot of people think of is oh well LLMs are just going to do the translation. What we found is actually by using LLM technology and classical neural machine translation technology with human in the loop, you get the best output. Actually instead of having the LLM do the translation instead of MT, if you have the LLM, if you do a machine translation and then you ask the LLM to fix it, you say here you go, now your job is to edit it, revise it, make it better, you end up with a measurably better quality outcome and then your human in the loop is more efficient. If they used to have to take it from a four or five to ten, now they’re taking it from six, seven, or eight to ten. The human’s more efficient and you’re using the human’s time more valuably and you’re also coming up with a higher quality output along the way. That is of value to customers. One of the things it does is it lets them ship marketing content more quickly. We have an example of a client where it was a 50% increase in their speed to market with some of their branded content and some of their marketing content and stuff like that. That’s a material ROI for the business. Would we have lost that client otherwise? It does enhance our offering and our competitive positioning. Yeah you surely expect it to manifest in revenue over time. But there are a lot of different ways to look at that and I just think that’s important too. Thinking about AI value in ways that aren’t just strictly about cost reduction. That’s not to be afraid of efficiency, it’s that it really can manifest in a lot of ways and might even require you to redefine quality and rethink what your benchmarks are. We’ve been working with Duke University, our alma mater, on setting new benchmarks for what translation quality looks like. That’s part of the game too. That’s part of the foundational element of how you end up measuring things.
Ross Katz: Yeah, that makes a lot of sense and a couple of ideas occurred to me as I was hearing you talk. One is that when you’re talking about people-based processes and the way that people are using AI, the intangible benefits proceed the tangible benefits. Part of being a great business leader is being able to understand what the intangible benefits are and accept them as a valid way of measuring things because sometimes measuring things as rigorously as you would like to measure them requires more resources than you’re really willing to invest in the measurement artifice. But then there’s other situations where when you’re talking about the integration of LLMs with neural machine translation, you’ve already got this business process, you plug the LLM in there, you can see immediately — you’re already keeping track of the quality scores of the things that you’re seeing at different phases, so you can immediately see the tangible benefit of that, but that’s because you’ve already set up the measurement interface, the measurement architecture.
Matt Sekac: That’s right. The whole foundations question keeps coming up because it’s important. If you’re a data team, you’ve got a responsibility too. You need to see what’s going on. There’s every reason to anticipate that this kind of stuff is going to become more and more important and more and more of a priority in lots of different organizations. While it is really important that there be alignment and a reasonable point of view around what ROI is but then also what it’s going to take to deliver that ROI — I think it’s okay for me to say Ross that we worked with your organization in large part because we were thinking proactively about what we need our data infrastructure to look like in order to be positioned for all of this stuff that we want to do. It might not be an emergency now, but the direction we see ourselves going, what do we need to be doing as a data organization to get us better positioned to scale with all of this ambition that the company’s got? That’s a data team’s responsibility, but there also needs to be, I don’t want to say patience exactly but an understanding that some of these more foundational investments might need to be made either at the same time as or before — sometimes maybe you could do them together, but they might be just absolutely necessary to actually achieve the kinds of things that you want to have live data connected to AI systems. Okay, what does live mean? We have to establish that. Are our systems set up to deliver if that’s hourly, is that every minute? Does the R&D team — are they equipped and do their structures allow them to actually take the data even if it is available at that cadence, at that pace? Everybody’s really got to be on the same page on all of this stuff.
Ross Katz: Before I let you go, what’s one thing brewing in the data and localization space, or just in the industry at large, that you’re watching right now?
Matt Sekac: For us — look, we’re watching everything because everything’s changed and turned upside down it seems like every three to five months. But just the way that these content generation and distribution pipelines are evolving and the technology being used to augment and supplement them. Obviously we’re developing a product now that we think plugs and integrates well with what a modern and future-state content distribution pipeline looks like and allows you to incorporate these capabilities with human in the loop and with technology and with some pretty cutting-edge technology to get the outcomes that you want. But the way that customers’ attitudes about those things evolve, the way that content is generated in the first place, and then the ecosystem of localization technologies evolves too, because there start to become opportunities for companies that provide enablement tools to actually become more like competitors. Then there also are situations where those enablement tools can actually be displaced by creative services from traditional service providers. LLMs are ultimately about language and text and that’s our business, so the stuff that we touch and the way that it flows to audiences is evolving. That’s something we really have to have our finger on pretty constantly.
Ross Katz: On my end, there’s this great article in New York Times today, this is Monday, February 2nd, where they bring together a bunch of leaders and people I respect in AI to talk about predictions for the future of AI and Nick Frosst, the CEO of Cohere, had this quote that AI will become boring in the best way. It’ll fade into the background like GPS or spreadsheets powering everyday tools and humans at work. It’s the most banal use cases that have the most transformative impact. That really resonated with me because I feel like what we have is this diffuse uneven intelligence that we’re trying to apply in all of these different ways and the uneven nature of it is tantalizing, it’s exciting, it makes you believe that anything is possible, but the diffuse nature of it also means that it can just be applied in so many places and in so many different ways. The number of ideas you can have about the applications for generative AI in particular and broader machine learning and AI in general is really fun to think about. Things like project management that you were discussing earlier, those are really some of the banal use cases that have the most opportunity.
Matt Sekac: The way I always think about it, or the thing I like to say, is it actually is a magic wand. You still have to figure out the spells. That takes a lot of work. You still have to figure out what to do with it. It doesn’t have an instruction manual.
Jason: Thank you, Matt. That’s it for this episode of Eventual Consistency. If you want to talk about your data challenges, or you think we got something wrong and want to tell us how, you can find us at corrdyn.com. That’s C-O-R-R-D-Y-N dot com. We’re doing this every two weeks. We’ll see you next time, where maybe we’ll get a touch more consistent. Thanks for listening.
