Listen on
Modern engineering organizations constantly grapple with legacy systems, complex infrastructure migrations, and the pressure to respond to incidents efficiently across diverse tech stacks. The promise of GenAI often feels distant or overhyped for these practical, day-to-day challenges. However, GenAI offers concrete, immediate utility in platform engineering and DevOps by streamlining tasks, accelerating troubleshooting, and supporting major modernization efforts.
Matt Barlow, CEO at StratusGrid, brings hands-on experience deploying and integrating GenAI into critical cloud infrastructure workflows. His team manages complex AWS environments for numerous clients, making his perspective on GenAI’s real-world impact invaluable.
James Winegar and Matt discuss how GenAI augments engineers with varied responsibilities, from debugging container builds to aiding the non-trivial migration from declarative Infrastructure as Code (IaC) to code-driven frameworks like Pulumi. They also explore the practical development of tightly scoped AI agents for incident response and the critical risks associated with over-reliance on these tools.
Key Takeaways
GenAI provides immediate productivity gains in complex engineering environments.
Platform engineers often juggle a dozen projects daily, working across multiple codebases, scripts, and APIs. GenAI, especially through chat interfaces, helps teams quickly write scripts, troubleshoot unfamiliar issues, and diagnose dependency problems in container builds. This significantly reduces time spent on common roadblocks, allowing engineers to focus on higher-value work.
Transitioning to code-driven Infrastructure as Code (IaC) significantly improves maintainability for large organizations.
As engineering organizations scale, declarative IaC languages like Terraform DSL hit limitations, particularly when hundreds of developers consume shared modules. Shifting to code-driven IaC using languages like TypeScript with Pulumi allows for robust type-checking, preventing invalid configurations earlier and creating more extensible, maintainable platforms. GenAI can accelerate the team’s learning curve for new languages like TypeScript, easing the migration burden.
Effective AI agents require tight scoping and custom tooling, moving beyond broad general-purpose approaches.
While general software engineering agents can be costly and inefficient due to repetitive errors, building agents tightly scoped to specific tasks—like gathering incident data or auditing IAM roles—delivers practical value. Integrating custom tools, such as a service registry lookup, empowers agents to retrieve context-specific information, turning them into highly effective assistants for defined workflows. This focused approach reduces operational overhead and improves reliability.
Over-reliance on GenAI without deep understanding introduces significant operational risk and hinders skill development.
Juniors, especially, face the temptation to use code completion or regex generation without fully grasping the underlying logic. This can lead to critical errors, security vulnerabilities (due to elevated access in platform roles), or outages. Prioritizing understanding and using GenAI as a learning aid rather than a replacement for knowledge is crucial for mitigating these risks and fostering genuine expertise.
Related: CorrDyn provides deep expertise in data engineering and technology strategy. We also assist clients with digital transformation initiatives and data cost optimization.
Full Transcript
Jason: Welcome to Data BS, a show dedicated to tackling the big questions impacting the world of data and ML AI without any of the BS. My name is James Winegar. Each week I sit down with guests from across the data ecosystem to unpack how they’re shaping their businesses or the businesses of others through their real world application of data engineering, ML AI, infrastructure, analytics, and more. No fluff, unfiltered, but slightly edited to remove noise, this is Data BS, let’s get into it.
James Winegar: All right, Matt. So, tell me who you are and what you do.
Matt Barlow: Hey James, Matt Barlow and I work at StratusGrid. I’m a cloud infrastructure lead. We have a product called Stratusphere, which is a multi-account FinOps product that helps you optimize your cloud costs. And then we also have a consulting segment of the business and that’s what I work in. I’m a long-term consultant engineer for platform engineering teams. That’s pretty much it for me in a nutshell. I’ve been working with cloud, AWS, DevOps-related type work for many years now. Really excited to be here and to talk about how AI is influencing some of the work that we do on platform and DevOps type of teams.
James Winegar: We’ve worked together on a few customers and that’s how we know each other, and you’ve really taken GenAI into your workflow, so that’s why we’re talking — to talk about Matt as a DevOps platform engineer, etc., the wearer of 20 hats. I know that y’all are an AWS partner, so there’s a lot of Bedrock going on for the stuff that you’re doing, but what specific GenAI tools are you integrating into your workflow that have impacted your day-to-day productivity?
Matt Barlow: For the agent type of work, I’ve been using Claude Engineer and Claude quite a bit, mostly on Bedrock and directly from Anthropic APIs. But for my day-to-day work, I’m really using a mix of different models, so I’m using the O1 preview, GPT-4, Claude on Bedrock, and Claude from Anthropic and other tools that are wrapping those models.
James Winegar: From a compliance viewpoint, you’ve made sure that all those tools are approved, and for the vast majority of the work you do, you’re not really sending customer information or data over anyway, it’s just, how do I do this AWS command line call and help me write a little program that does this stuff?
Matt Barlow: Yeah, for that type of work, everything is pretty much sanitized. That’s the way I like to work. I’ve played around with a lot of the code completion tools, I’ve used Co-pilot, I’ve used Cursor. But especially for customer work, I’m not using code completion, even though those tools are approved at some of the places that I work. Mostly the way I like to use the chat interface is if I’m stuck on something, to come up with a generic question and get a response back to help me move forward. That’s the way I’m using it now. For people that want to actually use GenAI for agent-type work or things that are going to involve business-related data, then Bedrock is definitely the way to go.
James Winegar: At least within the environments you’re working in, because it’s got your compliance approval, it’s not leaving your data ecosystem, stays within VPC plus-minus.
Matt Barlow: Yeah, it could stay within the VPC and you can fine-tune the models, which creates a copy of the model in your account as well that’s just dedicated to you.
James Winegar: So in your experience, what are the main efficiency gains from using GenAI in the DevOps or platform engineer role, particularly with processes like containerization or infrastructure automation?
Matt Barlow: I would say the first efficiency gains are, you alluded to it earlier, that when you’re working in these types of roles, you’re working with many different codebases and a lot of different scripting and APIs, and in a given day you might work on a dozen projects and different types of tools. So the first thing is, GenAI can help people write the scripts and troubleshoot things that they may not be working with every day. That seems to be important. But in terms of taking it to the next step with things like containerization, it’s really helpful in being able to diagnose build issues and help you get past dependency problems or other issues that you might be facing when trying to work on those things. What I’m really excited about, and I think we’ll talk about in a little bit, is how it can augment your automation in terms of trying to modernize or to go to end-to-end containerization of an application. There’s certain things that can be automated pretty cleanly, but there’s other things that require some human intervention, and I think GenAI has a lot of opportunity to help us with those. Tech companies that have been around for a while, especially if they have larger engineering organizations, there’s always going to be some sort of modernization effort going on. This particular customer has over 1,000 GitHub repositories and over a dozen teams that are constantly churning out new services and microservices, and what’s going to happen over time is that there’s legacy patterns. Working on a platform team like I do, what we want to do is the best job that we can in getting everybody onto consistent tooling and patterns. So in terms of CloudFormation and Terraform and Pulumi and so on, I think that’s the evolution of a platform, the direction that we’re going in DevOps and platform-related teams is now we sort of think of the platform as a product and the development teams are our customers. So we’re really thinking in more terms of product development, and using a real programming language, like with CDK or Pulumi, is going to enable you to develop or to create a product that is going to be more maintainable and extensible and avoid some of the annoyances with Terraform.
James Winegar: So really, it’s not just about the tools that we thought about, it’s the transition from a declarative format into a code-driven format where you’re writing TypeScript or something like that. Because you have this actual programming language that’s defining everything, you can create modules that are much more extensible than they were with a DSL. DSLs were really good for getting started, but as the scope of the operation has increased, the need for these extensible patterns increases. DSL’s a great place to start and then as your organization evolves, getting into more of a code-driven infrastructure-as-code framework.
Matt Barlow: Yeah, just to give you a couple specific examples, this really comes into play when you’re dealing with many, many consumers of the modules that you’re creating. Typically on a platform team, you’re going to be writing automation and the reusable modules that people are using to build their services. Once you have hundreds of people using that module, then you start to hit some of the limitations of things like the declarative languages that you’re talking about. For example, in Terraform, if you’re writing a Terraform module and you want to only allow a certain combination of inputs, that’s difficult to do in Terraform and typically what you see people doing is adding comments and things like that. But when you’re using something like TypeScript, you can do interfaces which only enable certain combinations of inputs into the module. So you prevent people from getting to the apply stage, and they can see in the editor as they’re using the module that something’s going to be an invalid combination. That’s one reason why the switch to TypeScript and Pulumi has been beneficial for us in a larger engineering organization.
James Winegar: Yeah, and I just happen to know there was hundreds of modules that y’all supported and things like that. That’s a very non-trivial effort in terms of the platform engineering team to go through that migration. I’m not sure it’s even 100% done still, it’s mostly done now, though. So how did you and your team leverage GenAI in that process of migrating? There was still some CloudFormation leftover when you started on it, so you have this CloudFormation and Terraform DSL and you’re migrating that into Pulumi’s development kit?
Matt Barlow: Yeah, I would say that GenAI for all of the people on the team, the greatest lift that it gives us is helping people get familiar with TypeScript. For a lot of people that work in roles like I do, typically you’re using Python a lot and Go. TypeScript is more of a full stack. Unless you’re doing back-end engineering, you typically don’t have a lot of exposure to it. I think that GenAI is really helpful in helping you to get up to speed on a new language. In terms of the modernization effort itself, we tried to automate as much of it as possible because we had dozens of services to migrate, and we use Backstage as well, so that kicks off our automation process. We have these modules that we’ve written in Pulumi, we can create a pull request on a service repository and even run the Pulumi code itself and build a lot of the infrastructure. But one place where we kept getting stuck was on the containerization phase. There’s really two steps that are difficult to automate. The first part is once you have the repo created, you can make your best guess at a Dockerfile, but it’s probably not going to build on the first try. And then the second part is when you actually go to deploying the containers, there’s going to be debugging that goes on. So I was really interested to know if GenAI could assist with those, because when you can automate pretty much the modernization of a service from end to end but you have two points that are semi-manual, I was curious if GenAI could help with that. We didn’t actually use it on this project, but it did lead me down the path of trying to evaluate agents for that scenario. So what I did was started working with some of the software engineering agents that are out there and tested out containerization on about six different code projects that were in different states of working, and I learned a lot from doing that and was able to finally get to the point where the agent could reliably containerize these six different applications.
James Winegar: Yeah, could you talk about what the differences between these six were, because it sounds like you picked six because they were different enough. Was it like a Go app and a TypeScript app?
Matt Barlow: Yeah, different languages and different types of projects. One of them was a full stack front-end and back-end application, the other ones were services, and then there were a mixture of languages and different things that were broken about them. One of them had a dependency issue, so I wanted to see if the agent could actually solve some of the issues that we would see during an actual migration.
James Winegar: Did it manage to get through that?
Matt Barlow: Yes, it did manage to get through it. But this is where it gets interesting because what I found was using the software engineering agents, they’re designed for a very broad use case where you can pretty much just ask it to build or do anything. I was really excited the first time that I saw it starting to run shell commands, but your bill’s going to go up really quickly because it’s going to be making a lot of the same mistakes, and it’s sort of like watching somebody and you’re thinking, wow, took you 10 steps to do that, but you could have just done it in two. I also noticed that they were making the same mistakes over and over again. That’s what led me down the path of trying to update the prompt templates, which in the tools that I was using at the time, required modifying the source code because there wasn’t a way to actually change the prompt templates or the sandbox environment that it was running in. I’ve moved past that and I’ve got a better way of doing that now. But that’s what I learned in that, and I started updating the prompt templates and that’s what enabled it to get past things like the peer dependency issue. Do I know 100% was I over-optimizing it, maybe I was, but it was an interesting project and I learned some good stuff from it.
James Winegar: We’ve talked about general purpose GenAI — ask a question, get an answer, use that to help get you unstuck in a piece of your workflow because at any given day you’ve probably got 10, 15 different things going on. How do you see the role of these AI agents evolving in DevOps? Because that’s where you can actually get some of that iteration cycle in, and then also these manual tasks that we were talking about with containerization?
Matt Barlow: That’s really what I’m interested in next, is building agents that are tightly scoped to particular use cases. It reminds me of when we started working with cloud, for example, and creating scripts and things for repeated tasks that we had to do. There was a place where I was working where even before the cloud came out, we were managing tens of thousands of Red Hat servers and a lot of them were running Apache, and every time we logged into a server, it was a new environment and we would be running the same commands over and over again. So we started writing scripts so that we could take the tribal knowledge and put it together, and then every time you logged in it would run through these 20 or 30 different checks. I see a lot of opportunity for AI agents in the same way. This gets back to what I was talking about earlier in terms of being able to tune the prompt templates. When you’re working with the agents, I think it’s important to be able to have these templates for different types of tasks and to be able to combine maybe some procedural code with the LLM-related code, and also to be able to have some control over the sandbox environment. Let me talk through that a little bit more. What I’m getting at is if I’m working on something that’s an incident, I’m going to be following the same kinds of steps every time. I’m going to be checking Datadog or New Relic, I’m going to be looking at AWS resources, I’m going to look for any recent commits that happened, what was pushed to production. What’s interesting is that I may be on a call with other people that do the same job that I do, but we may all approach it differently and we may forget a step. Where I think this starts to become powerful is that we can have an AI agent where we know that we’re going to need these 10 pieces of data from all these different providers. Can we just write a script that grabs that at once and then maybe summarizes it, looks for insights and provides us with some data to help us with incident response. That’s the direction that I’m interested in going. I already do this for a lot of ad-hoc scripts and things that I need to write. For example, if I’m asked to do an IAM role audit, and there’s thousands of IAM roles and I need to audit the policies in each one, rather than just opening up my editor and starting from scratch or even using something like ChatGPT, I have an agent that has a template that states exactly how I like my scripts to be written using TypeScript, being executable with the Bun runtime, and every other instruction that I have in there so that I could just tell it what I want and it creates the script exactly the way that I want it. This is where I think the real power starts to come into play — the ability to set those prompt templates and to combine it with things like if you wanted to do a pull request review, that’s procedural: grab the data related to the pull request and then do the analysis of it. That’s what I’m excited about.
James Winegar: One of the things I’m thinking about on the client that we both work on, one of the reasons I get called in is the shit’s hit the fan on a database. And almost always I’m like, go grab these five pieces of information, and based off the outputs of those five, I’m going to give you two or three other ones, or I’m going to be like oh it’s this.
Matt Barlow: Something related to that that I’ve been playing with the past couple weeks is having custom tools that are designed for this work. Typically, the agent that I’ve been using recently is Claude Engineer, which has been a really great project to work with, but it’s not designed in a way that’s really extensible, it’s not packaged or anything, but it’s really easy to clone and get started with and make changes. All of these agents have a lot of tools like Tavily search or a bash tool. But what I’ve been interested in lately is writing tools specific to infrastructure work. The first one that I wrote was a service registry tool, which basically told the agent if you need information about a service or you need to list all services, then call this function. So then I could ask the agent questions like, what is the logging command for the ECS task for service ABC? And then it could pull the metadata out for the service and do a resource lookup and get me back the ARN and other information that I need to run that command. What you’re talking about, I want to keep exploring. It would be interesting to have these tools related to data as well for the agents to say, how do you check resource utilization, how do you check IO, so that the agent can make decisions in terms of what tool it wants to use in order to get the data that it needs.
James Winegar: Yeah, I’m less hype on agents because I think that it requires so much context and that context is usually very dependent, so to your point. You’re writing these functions that go call into your service registry to extract some metadata. That required you to have a service registry — is the service registry for your client going to be the same as some other client or some other company? Probably not. You could say, oh this works for AWS, but it might not work for Azure, this works for Azure and it might not work for Google Cloud. It’s very sub-setting in terms of how it behaves, and then also chains of agents. If you really dig into a workflow and you’re like send an email — there’s a lot of steps in sending an email if you really try to break it down into what an agent needs to do. So there’s this planning, this meta-planning requirement that’s there. Do you have a separate LLM that is supporting meta-planning, but there’s often hallucinations, especially when you have this very structured thing that you’re trying to get to at the end of the day. Do you have approved sets of plans based off the workflow and then it’s a search engine against these approved sets of plans. Our viewpoint typically is the more focused of a problem statement you have, the more valuable GenAI can be. If you’re fine with sub-setting it to where it’s for this particular thing in this particular environment, it can work really well. But these general-purpose AGI, I’m not very bullish on AGI, I think we got a lot more compute necessary to get to that point.
Matt Barlow: Yeah, I think you’re exactly right. Any attempt that I’ve made to use agents to break down a problem and then create sub-agents and then they report back to the agent that’s tracking the project overall, it always seems to fall apart. I haven’t seen anything practical come from that. But to address your comment about the implementation of these different services for infra-related work, I think that’s a really good point. What I’ve been doing in Claude Engineer is I write the agent tools, but the actual functions that are being invoked are like a strategy pattern. Basically the function itself is implemented separately that actually pulls the service registry data, so that if you did want to implement it for different customers and different environments, you only have to work on the implementation piece — the agent code itself wouldn’t change.
James Winegar: Let’s switch gears because we’ve been talking about how you leverage the tools, but let’s talk about what is changing for people as a consequence of this tool. What are the risks of actually relying on GenAI, especially for juniors on their team, and how can your team mitigate these risks?
Matt Barlow: That’s a good question. The risks, two of them stand out. One of them is just the fact that usually when you’re working on something like a platform team, you have elevated access. So you have to be very careful about what you’re running and making sure that you understand it thoroughly, and you definitely don’t want an agent running shell commands on your behalf when you have credentials, keys to the kingdom. That’s something you have to be careful of, but aside from that, it really gets down to being able to understand what’s being produced before you implement it or run it. For example, if you are responsible for request routing rules, like for something like Cloudflare that makes heavy use of Regex, then it could be really tempting to use something like GPT to generate Regex rules for you. But do you really understand what’s happening? What if there’s another rule that shadows your rule, or what if you don’t understand how it’s being written and it could cause an outage? I think relying on it without understanding it is a big issue. The other issue that I see is people using it for code completion when they don’t understand the code that’s being completed. It can be so tempting because we’re all under so much pressure to deliver, and if you’re using something like Cursor and you’re tab completing through a bunch of code without necessarily understanding it, but you get it to compile or it works and it seems like it’s working, then you’re not really doing yourself any favors because you’re not learning how it’s actually implemented. I think that can be an issue. That’s why I prefer the approach that I mentioned earlier, which is if you’re new to something like TypeScript and you’re writing a Pulumi component and you don’t understand how to do something, then think about the question that you want to ask and make a generic request to something like GPT to get the response back and then fit that into your code, rather than just having the editor write it all for you.
James Winegar: For people interested in learning more about this stuff, can you share any recommended resources, blogs, projects, where people can see these types of things in action?
Matt Barlow: Like I mentioned earlier, I would recommend Claude Engineer as a good place to start for learning how to evaluate agents on your own projects to see if it’s working for you. What I’d recommend is basically cloning the Claude Engineer repository and using it as it’s designed, but then also customizing it according to different use cases that you have. I think that just getting started with it is about experimentation. When I started working on this, I was using LangChain, I tried writing my own custom software engineering agents. I also used a project called OpenDevin, and then lately I’ve settled on Claude Engineer, which does integrate with Bedrock. Using all these tools you’re going to find out what works well and what’s not working well. Really what it comes down to in my opinion is being able to customize the agents to your use cases, which I found Claude Engineer to be the best for that.
James Winegar: Have they added hooks now for Claude Engineer for you to override the prompts, or is that still a thing where you’re having to rebuild it based off the prompts that you’re doing?
Matt Barlow: They have not done that yet. But what I’ve been doing is basically just cloning it, and then I have a separate project and I’m able to make minimal changes to Claude Engineer to have a separate tools file that gets included. There are little Python hacks and stuff that you can do to get around the fact that it hasn’t been built in a way that’s extensible, but I still find it to be worth it.
James Winegar: All right, Matt, it was awesome to talk to you. Hope you have a great day. Good luck with everything.
Matt Barlow: Thank you, James. I appreciate it. Thanks for inviting me.
Jason: That’s it for this episode of Data BS. If you enjoyed this episode, make sure you subscribe wherever you listen to your podcasts and not miss the next one. This episode was sponsored by CorrDyn, a data consultancy that helps organizations unlock the power of their data. If you have a data challenge, we can help. Visit corrdyn.com, C-O-R-R-D-Y-N dot com to learn more. See you next time.
