Listen on
Organizations struggle to balance rapid data development with the demands of large, growing datasets and tight compute budgets. Traditional data transformation tools often force engineers to choose between slow, full refreshes for data integrity or complex, error-prone manual incremental updates. This choice directly impacts development velocity, data freshness, and cloud spend.
Toby Mao, CTO of Tobiko Data and a veteran of Netflix and Airbnb’s data platforms, joins us to discuss how SQLMesh changes this equation. Drawing from his experience building systems at hyperscale, Toby reveals how SQLMesh’s core capabilities—native state management and deep SQL parsing—eliminate these trade-offs.
We explore how SQLMesh enables true incremental processing, prevents data gaps, and provides instant, cost-effective development environments. Toby explains how compile-time SQL validation and virtual data environments accelerate developer cycles, reduce errors, and allow for instant, low-risk deployments and rollbacks. The conversation also touches on how SQLGlot, SQLMesh’s underlying engine, facilitates multi-engine support and significantly reduces data warehouse vendor lock-in.
Key Takeaways
Robust state management prevents data gaps and speeds up large-scale transformations.
Traditional stateless ETL tools often reprocess entire datasets or risk missing intervals when runs fail. SQLMesh’s first-class state tracking fingerprints transformations and records run history, ensuring only necessary computations occur. This guarantees data completeness even after interruptions and significantly reduces compute waste.
Deep SQL parsing eliminates wasted compute and identifies critical changes early.
Most data tools treat SQL as opaque text. SQLMesh, built on SQLGlot, understands SQL syntax and logic, allowing for compile-time error detection before any execution. This intelligence also identifies non-breaking schema changes to avoid unnecessary backfills, minimizing cloud spend and development cycle time.
Virtual data environments enable risk-free, instant data deployments and rollbacks.
Data developers typically work on non-representative data or costly copies. SQLMesh’s virtual environments create instant, view-based copies of production data for development and testing. Deployments become near-instant pointer swaps, meaning production remains stable even if a new version fails validation, a crucial operational safeguard.
SQL transpilation delivers true data warehouse independence and multi-engine flexibility.
Vendor lock-in remains a major concern for data leaders. SQLGlot’s ability to translate SQL between dialects allows models written for one warehouse (e.g., BigQuery) to execute on another (e.g., DuckDB). This capability supports cost-optimized multi-engine strategies, local unit testing, and simplifies complex data platform migrations.
Related: CorrDyn helps organizations with complex data engineering challenges, focusing on data reliability and data cost optimization. Explore our insights on modern data tools like SQLMesh for improved efficiency.
Full Transcript
Jason: Welcome to Data BS, a show dedicated to tackling the big questions impacting the world of data and ML AI without any of the BS. My name is James Winegar. Each week I sit down with guests across the data ecosystem to unpack how they’re shaping their businesses or the businesses of others through the real world application of data engineering, ML AI, infrastructure, analytics, and more. No fluff, unfiltered, but slightly edited to remove noise. This is Data BS. Let’s get into it.
James Winegar: Thanks for joining us, Toby. Can you tell us about yourself and how Tobiko came to be with SQLMesh and SQLGlot?
Toby Mao: Sure. Thanks for having me, James. My name is Toby. I’m the CTO and one of the three co-founders of Tobiko Data. We make SQLMesh and SQLGlot, two new-ish tools in the data transformation game. I got my start in data doing a lot of data work at Netflix, where I led the experimentation platform team. I worked a lot on big systems like Spark and Presto, building out data science and analytics platforms there. After that, I moved to Airbnb, where I started to work on Minerva, their metrics platform, as well as their experimentation platform. I saw what it takes to build a data transformation platform that can work at a company like Airbnb, and decided to give it my own shot to take what I’ve learned and build something for the public.
James Winegar: The similar ideas that you had to do from scratch internally, you were able to apply and put towards an open source framework where you’re trying to take all the best approaches that you thought really needed to be there.
Toby Mao: It’s definitely the best of all approaches. A lot of the things that are done at FAANG companies might not apply to most people in industry. For example, one of the things that we did when we first started SQLMesh is that we thought that our primary audience would all be using Spark and Airflow. That turned out not to be true. Most people are using things like Snowflake and BigQuery and Databricks. As we went further along with building out the product and the company, we learned a lot more about what most people are looking for.
James Winegar: People are victims of their experience and when you don’t see a thousand different things, you got here’s the three or four places I worked, it was all very heavy tech, and then learning what is most used and then identifying your ICP from there.
Toby Mao: Exactly.
James Winegar: What was the real true trigger to spin up SQLMesh for you? I’m guessing something happened at work?
Toby Mao: I always wanted to start a company. Initially I was thinking about building out a semantics layer because of the work I had done with Minerva. But then I realized that the true power of Minerva wasn’t the semantics layer. The semantics layer is cool, but Minerva was really useful and powerful because it was a transformation framework. While I was there, I got a demo of dbt and I just started asking a couple questions. Well, it looks like you’re just refreshing the database every time you do dbt run. How does that work? It’s like, Oh, well, you can just refresh everything. I’m like, well, that would never work here. We can’t do that. Or what if I just want to run one day of data or this time range of data? It’s like, you don’t do that. You just run everything. From there I started doing a little bit more digging and I was like, well, this tool seems to be pretty popular and there seems to be a need for data transformation tools. Why don’t we try to build something that can scale at any company and be just as powerful? That’s why we started it.
James Winegar: Identifying the gap in capability and then build out from there pretty much.
Toby Mao: Exactly.
James Winegar: I think that transitions pretty well into a high-level outline of why incremental processing is necessary when you have large scale, maybe from a time series viewpoint or something along those lines.
Toby Mao: Right. dbt is really great when your data’s simple and you can have a fully stateless warehouse. If your data’s small and you don’t have that many models and you’re able to just either make everything a view or fully refresh everything, it’s really nice because it’s very simple and you don’t have to worry about anything. You just do dbt run and everything’s good, everyone’s happy, right? But working at larger scale, especially at a FAANG company, you can’t do that anymore. There’s just too many tables and the tables are too big. Instead of refreshing, for example, the table that tracks every single video play at Netflix, you have to start processing things every day, one day at a time or even every hour, one hour at a time, because each day might have petabytes of data and you can’t just refresh years of petabytes of data. Once you start to incremental processing, it gets a little bit more difficult. Because there’s state involved. You want to know which dates or which intervals have been processed. You need to keep track of that. And you want to only do what needs to be done. One of the core differentiators of SQLMesh is that it has first-class understanding of state. Every time you run something, it fingerprints and tracks that, as well as it stores when you ran things for and then what needs to run next. Every model in SQLMesh also has a concept of cron. It’s got built-in scheduling capabilities as well as tracking of when those things need to happen and what has already happened.
James Winegar: And then you’re saving that into the warehouse itself, some state table, and then that way you can reference it or
Toby Mao: Exactly. By default we wanted to make SQLMesh work out of the box and so you use the same warehouse for state. For example, if you’re using BigQuery for your engine, we store all this extra metadata into the warehouse. But warehouses aren’t really designed for transactional processing, things like for every run, keeping track of the state or whatever. You can optionally set up a Postgres database, which we suggest, or you can use our new cloud offering which has hosted state.
James Winegar: Which is a Postgres database under the hood.
Toby Mao: Exactly, a Postgres database.
James Winegar: Is it only Postgres that supports for the state management layer or do you support other things for that?
Toby Mao: We support other first-class transactional stores. I think some people use MySQL with it, but most of our testing is done on Postgres. But MySQL should work.
James Winegar: Postgres just works for the vast majority of these things and the only reason people would use MySQL is they’ve got a thousand other MySQL databases and they don’t want to have to think about the difference between Postgres and MySQL for whoever has to manage it.
Toby Mao: Well, to be fair to MySQL, I believe it has better sharding capabilities and companies like Uber, I’ve read that they choose MySQL because they can heavily shard and have multiple databases.
James Winegar: One of the reasons I really like SQLMesh in particular is compile-time checks for your models. Could you talk to what, besides incremental processing, really makes SQLMesh support the developer efficiency more effectively?
Toby Mao: One of the big value ads of SQLMesh is that it’s built on top of SQLGlot, which is an open source SQL parser and transpiler that I built a couple years back. What that does is it gives SQLMesh the ability to actually understand SQL. Traditionally many other ETL tools or data transformation tools, they don’t really understand SQL. They just stitch together SQL with Jinja and then they just send it to the warehouse. It’s up to the warehouse to decide what’s right or wrong. With SQLMesh, we took the opportunity to build it from the ground up to understand SQL. When you actually do a SQLMesh plan, which is similar to a run in dbt, basically it compiles all your SQL and understands it. If you have a syntax error, like you’re missing a parenthesis or something like that, or you don’t pass in the right argument into a function, it’ll come back and say, Hey, this doesn’t look right. Instead of having to execute 10, 20 models and then for your 21st model to say, Hey, you have a syntax error, wasting all that work, SQLMesh will say, Hey, you broke this, fix it and then it will run. It won’t do any processing. But again, going back to state. In that example of the other framework, with dbt, let’s say you made some kind of error. Then you processed 20 models and then your 21st model errored. If you just did dbt run after fixing it, if you just did dbt run again, you processed all those models again. Because it doesn’t know what has been done. With SQLMesh, we catch a lot of errors but we can’t catch everything. Because ultimately it’s up to the data warehouse to be the source of truth. Even if that happens with SQLMesh, you get to the 21st model and then it errors, if you rerun it again, it’s only going to start from there. It remembers where it last left off.
James Winegar: Or it might run the next increment because it was an hour later and based off your incremental, so
Toby Mao: Ah, that’s another benefit of SQLMesh. I think dbt recently launched what they call batch incrementals, and that’s similar to our native incrementals. But because they don’t have a concept of state, if you miss an interval, let’s say you had a process at 12 p.m., but it’s been an hour so it’s already 1 p.m., you’re not going to process 12 p.m. You’re going to skip it, there’s going to be a data gap. With SQLMesh, when you run it and it’s 2 p.m., we’ll know that 12 p.m. and 1 p.m. haven’t run yet. We’re going to batch all those up and run all those together. You never have to worry about manually defining what needs to run.
James Winegar: And then in that you’re going to run three different models, the 12 p.m., the 1 a.m., the 1 p.m. and the 2 p.m. increments.
Toby Mao: There’s so many differences. But another difference with SQLMesh is that it can batch them up. Instead of having to run three separate jobs
James Winegar: Oh, so dynamically, taking the windowing time as long as you don’t set a maximum threshold or something for volume reasons, right?
Toby Mao: Exactly, exactly. You can control that as well, that’s a toggle. But by default we would just run one job from 12 p.m. to 2 p.m.
James Winegar: And then you’re watermarking that in your state, basically. I learned something new now. I would have figured it would have run three different jobs to manage the interval. If you have a failure due to, I don’t know, let’s just say you’re running on an extra small Snowflake or something like that and it crashes out for whatever reason, like timeout, because you have a 10-minute timeout on the query or something, will it try to shrink in the interval time?
Toby Mao: Not right now. There’s so many different things that can go wrong. You don’t know if it’s you, kind of thing. As of now you could just change your batch size. We call it a batch size, and you can make that, for example, one hour instead of ten hours. Then you’ll only process one hour at a time.
James Winegar: And then you guys can group it based off the toggle that we were talking about before.
Toby Mao: Exactly. Yep.
James Winegar: Awesome. Is that like maximum number of batches or something like that is the toggle?
Toby Mao: We have a concept called interval unit. That represents the grain, time grain, of your model. By default it’s daily. If you say that you have a batch size of 10, that means every time you run you can run a maximum of 10 days per job.
James Winegar: Depending on whatever you defined your grain, so it’s the interval multiplier by the grain basically. Okay. And then because you manage state, you can handle that dynamically pretty much.
Toby Mao: Exactly.
James Winegar: What other things come to mind for you?
Toby Mao: I would say an innovation or differentiator of SQLMesh is what we call the virtual data environment. We wanted a way for SQLMesh to enable developers to very quickly be able to work on representative data. At Airbnb, Netflix and basically every company that I’ve worked at, developers develop on production data whether or not you like that because, for example, if you’re writing a machine learning model or you’re trying to compute statistics or analytics, doing that on fake data just doesn’t really work because you don’t know what you’re working with. It’s not representative. Traditionally at these companies it’s hard. Because you have to manually point to the right tables in production, make a copy, it’s expensive. Because we have state, because we can track things, we know what the definitions of models are. When a developer in SQLMesh creates a new development environment, basically we can do automatic creation of the development environment without doing any work. The way we do that is through views. SQLMesh has two layers. It has what we call a physical layer and a virtual layer. The physical layer is basically machine-managed. You don’t have to worry about it. Basically it’s a bunch of hash tables. That’s where all your actual materializations go. Your tables, your views, they all go into there, but they’re hashed with a special fingerprint. On top of that we have the view layer. Your actual prod environment or your dev environment are just all views and they’re just pointing to one of the physical tables. When you’re creating a dev environment based on prod, you don’t have to do any processing. It just creates a bunch of views mimicking prod pointing to the right tables. When you make a change in dev, we’ll understand, Okay, you changed this model and these dependencies. It will just create the physical tables that are necessary and then repoint all those views to the physical tables.
James Winegar: For the dev one, right?
Toby Mao: For the dev one, that’s right. Then finally when you’re ready to go, you’ve double-checked everything, CI/CD is good, green, when you want to deploy it, again, you don’t have to recompute anything like you would with other frameworks. We would just in prod swap all the view pointers that have changed to the new physical tables. You can get instant data deployments and even instant rollbacks.
James Winegar: Because under the hood you’ve already done the work. Because you’re operating on the production data set, because that’s the only way you’re really going to be able to be like, oh yes, this piece of information makes sense and you’re using dashboards or something, you do a preview deployment onto that and you can look at how BI got impacted. Okay, yeah, we’re good to go with that. Now I’m going to merge it. Well, I already got the data transformation I’ve done. Now I just change the pointer. This all backs out of you have the ability to keep track of state. Because you have the ability to keep track of state, there are other things that you can handle for the developer.
Toby Mao: That’s exactly right. Since it’s a huge efficiency gain, you can start building representative data. For the first time in data development, you can actually promote data that you’ve been working on to production. You can actually promote a table now instead of having to just promote code and then rerunning everything, you can actually just elevate a table to be used.
James Winegar: I got a different question then. What about pruning of the physical layer? Because I did a bunch of dev work, some of it to throw away I don’t need it because I messed up my window function, those types of things. How does SQLMesh help with the cleanup process?
Toby Mao: That’s a great question. Obviously as you have more development environments and you make changes, you’re going to have more of these physical tables. That’s why each physical table has a property called a TTL, time to live. That’s by default one week. Basically every time you run the janitor, it will check for any table that’s over a week old that’s not being used, and it will delete it.
James Winegar: Okay, so you’re just pruning in the background basically. Hey, this table’s got a TTL of whatever, so I’m going to and it’s been longer than whatever, it’s time to just truncate the table and delete it out.
Toby Mao: Right. And it’s not being used in production or anything like that. Because you have all state, basically.
James Winegar: Exactly right. I want to do a moving average for some column and I want to make that a new column. Let’s just assume we got a petabyte of data for giggles. I’m not going to be able to do that in a single batch or whatever. I have an analyst or a data engineer, analytics engineer, pick your person, they update the SQLMesh model to have this moving average transformation on, I don’t know, price or something like that. In the development environment, I’m going to do all that work. There’s a bunch of partitions there that it has to go over for each of the increments because the data set is large. Is there any work that happens where you’re able to say, Oh, I’ve already done that particular work in a different transformation somewhere bringing it out, because a lot of these data warehouses are column-based so I could pull out that column to use as a reference, but that also might be janky because that involves bringing together a join or something. Is that throw away for during the dev cycle, if you got these transformations that you’re working on and you have a larger data set, you still need to be careful is really where I’m trying to go with that about not wasting a bajillion compute units or whatever.
Toby Mao: Generally with data transformation tools, you want your code to be idempotent and you want your code to be self-containing. You don’t want to create a table with ad hoc SQL. You want to create a table with SQL that’s written in your transformation framework so that it gets checked in and all that kind of stuff, and then you can recreate it. When you’re working on something in dev with SQLMesh, you put it into a model and basically run it. A couple things that you can do to make it more efficient are one, first of all, in dev, you can just run one day or one time window at a time so you’re not processing too much data and you can test it and make sure that it scales. But two, there’s a couple of other things that we can do to make it more efficient. One is you can do a self-referential model. You can refer to yourself to rebuild the next computation. For example, if you’re computing something like lifetime revenue, instead of computing lifetime every single time, which is a full scan, you could just look at yesterday’s data and then from yesterday’s data you can then add today’s data. That’s something you can do. But the tricky thing with that is that now your model refers to itself. It can’t be run in parallel anymore. Because we can parse SQL, we can detect that automatically and then we’ll automatically switch your model to be sequential. That’s an example of another really nice
James Winegar: You don’t have to think about it as an analyst or whatever. Somebody else already knew that you would break it if you did that, so therefore we’re gonna flip this switch for you instead of you having to think about it. That’s a really hard problem, to minimize computation while also allowing for the thing to evolve effectively. Are you seeing with SQLMesh Tobiko customers, are you seeing even their dev cycles happening in CI/CD?
Toby Mao: Absolutely.
James Winegar: Yeah.
Toby Mao: We have an open source CI GitHub CI/CD bot. We see many, many companies using SQLMesh in conjunction with CI/CD. A lot of people when they do development, they do that locally. But in order to get something to production, you have to go through a PR process. When you make the PR, the bot automatically creates a PR environment. What’s nice is that it runs all the checks and runs all the evaluation. But then people can actually look at the environment to see what that data looks like, which is really nice. Once you get the required approvers and it passes the checks when you merge it, the bot automatically also promotes that environment to production.
James Winegar: Because you have state management, you’re able to manage that whole workflow because it’s just another virtual data environment. You do the incremental transformations or whatever you need to do. You might have to do some history based on what it is, and then just go through the merge process that you have. Yeah.
Toby Mao: We’ve worked with a couple different companies that used to use dbt for a long time. One of the amazing benefits that we’ve seen is that you actually get a true data deployment. What does that mean? With dbt, usually how it works, you do a PR and when you merge the data in, that’s now in production. When you merge in your code in the main branch, your next dbt run’s going to kick off and it’s going to run stuff. Now if you broke something, your production is now broken because when you do a dbt run, it’ll crash. But with SQLMesh, you can’t change the production deployment until it actually runs. Even if you were to mistakenly merge something into production with broken code, if it crashes, it’s not going to affect production. Production will be on the previous version that still ran.
James Winegar: It’ll just be stale or something like that.
Toby Mao: Exactly. It’ll just be stale. For the first time, you’ve got a real data deployment, which is pretty powerful.
James Winegar: Instead of having to do some of the things we’ve done before is I’m gonna have a step where basically I build everything into a particular database or a particular schema or something like that and then the last stage is actually a promotion lifecycle management, but that’s complicated because now you’re managing a bunch of state in SQL which is not exactly the best place to manage state. There’s always something happens, can’t cover every edge case in what I am talking about and so by forcing the run completed successfully promotion cycle, you don’t run into a place where let’s say you have data test cases that start to fail because new data comes in that’s not meeting expectations, well you can keep your production environment just stale while you try to do an investigation. It’s not even something that’s dependent on the deployment process, it’s also dependent on your data verifications and things like that that you do. Because in a lot of cases for us, what happens is there’s an engineer or software product doing something, they make a bunch of changes, they change fundamental ideas about the data, maybe they do a big migration in the application layer, nobody told the data team that hey we changed the way that this compound ID is actually generated and so your split logic or something like that doesn’t even make any sense anymore and then that ends up in your warehouse and then suddenly your test cases aren’t passing. Well then you don’t actually promote on that run to production. Am I understanding that right?
Toby Mao: The deployment process can be done in different ways, but basically first you gotta make sure your code compiles, all the SQL’s good. Then you gotta make sure that the code actually runs, and then once that’s done, once you’ve actually created those data assets, then you can promote, swap all the pointers. All this is done, again, with representative data. You might be able to build a custom thing with dbt to do this, but you’re probably going to do it in a different target warehouse, or different target database. Again, it’s not really representative data and just because it ran in your target database doesn’t mean when you actually merge to main it’s going to run on your production database. There’s just a lot of gotchas to be had if you don’t have virtual data environments.
James Winegar: Is there any other particular things about SQLMesh that you think really impacts the dev workflow?
Toby Mao: Combining state, SQL parsing, a deep understanding of SQL, allows us to do really, really powerful things. SQLMesh has a concept called breaking and non-breaking changes. As an example, if you’re working on a table and you add a new column, nobody else uses that column downstream. That’s a safe change. SQLMesh automatically says, Hey, this is a non-breaking change. We can backfill this table that you added this new column because it’s new, but nobody downstream uses this column, so we don’t need to backfill any of those. You can see SQLMesh is smart and understands what happened and what needs to happen. It saved you a ton of computation because you didn’t have to recompute all the downstream things. Now if you were to change the where filter or add a join, that’s a completely different story because all the downstream models could be affected. Their cardinalities could have changed as well and we call that a breaking change. Understanding state gives us the ability to understand what has changed. Because state allows you to know what it was before and then when you change it, what it becomes. Understanding SQL lets you say, Oh, okay, you added a column here, it’s okay, or you changed the where filter, that’s not okay. Combining those two things gives us the ability to automatically only run exactly what is necessary.
James Winegar: It’s a minimization of the compute utilization as well. Also minimizing the iteration time because one thing that happens is you have sufficiently complicated project that takes two hours to run or whatever. If I only have a small non-breaking change, I really only need to look at this one table and that might be fairly easy to do relatively speaking in terms of compute time. Let’s switch gears to, you had already talked about SQLGlot a little bit. One of the things I really like about SQLGlot, which is independent of SQLMesh but also can be used within SQLMesh, is the fact that it standardizes the way you write SQL in a way that allows you to have a given model or whatever that’s written in Snowflake syntax or BigQuery syntax, but because SQLGlot understands SQL, it can transition that to DuckDB or Redshift or whatever other target that you want to do. It really eliminates a lot of vendor lock-in with data warehouses. I wanted to hear your thoughts about SQLGlot and how that ties in with different migration strategies that people might run and how SQLMesh really enables companies to be a little bit more independent of their providers.
Toby Mao: I created SQLGlot when I was at Netflix because at Netflix and many other companies like Netflix, they use multiple database engines. At the time they used a lot of Spark, a lot of Presto and Druid. Data scientists like to write SQL, but they didn’t want to write the SQL three times because Presto, Spark and Druid all have very different syntaxes. SQLGlot was created so that when someone writes a Spark query, we can actually transpile that and then run that in Presto. SQLMesh basically automatically gets that capability. If you have a bunch of models written in Snowflake or BigQuery, you can actually run all those queries in DuckDB. We leverage that ability to give you, for example, unit tests. In SQLMesh to save money and make iteration speed faster, instead of running all your queries against real data in the warehouse, you can run all of your queries against fixtures or fake data. Basically YAML files or JSON files of fake data and basically assert for expectations. You’re like, Okay, this query given this data set is going to return this data set. What’s nice about this is that it gives us the ability to run a SQL query in any warehouse we want. Instead of having to re-implement, for example, every warehouse ourselves, trying to actually run BigQuery code or having to run Snowflake code, we can just use a nice engine like DuckDB that does all the work for us. We just really have to translate the SQL queries. What we’ve seen some of our customers do is they want to avoid vendor lock-in. They love the unit tests, they run their things in DuckDB, but maybe a company, a warehouse, really starts increasing their bills or for some reason they want to switch from warehouse A to warehouse B. With SQLMesh, it gives them that flexibility because now they can just take all their models and direct them to a different warehouse and you’re good to go. That’s one part of the story. Just running the queries that you wrote in a dialect to another warehouse. But after that, you want to make sure all the data’s correct. Because there might be differences in types, maybe floats have five digits in one warehouse and four digits in another. SQLMesh also comes with data diffing. It gives you the ability to check whether or not two tables are the same or not.
James Winegar: Across gateways, which is the warehouse basically?
Toby Mao: Right. In the open source version, the data diffing is only available within the same warehouse. But in our cloud product, we offer cross-database data diffing. Basically we’re able to across databases efficiently compute whether or not the tables are the same, and we do that basically by computing hashes over the tables.
James Winegar: And then you fingerprint by ID or something like that. How efficient is that process?
Toby Mao: It works pretty well across really large data sets because we basically do a binary search over the tables using efficient deterministic hashing, and then we can pretty quickly and efficiently get you a sample rows that don’t match.
James Winegar: That’s a big deal. I’ve had to implement that myself 10 times for different customers, because you’re moving databases and you need to verify that your data, I’m just going to say SQL Server to a Postgres database, it’s the same thing. What you’re telling me is that if I use your cloud version then I could go against my as long as it’s a supported database, those types of things, I could verify from point A and compare point B and ensure that I have the exact same data in both spots.
Toby Mao: Exactly.
James Winegar: That’s a product by itself. Right. Maybe not a gigantic one because it’s got such a niche use case, but in part of the larger ecosystem, that’s a thing that would save a lot of time.
Toby Mao: That product exists. Datafold is that company that really specializes in this field, and I think they do a really good job in creating a nice UI and feature set around this capability.
James Winegar: Cool. I’ll look at Datafold when we’re done then. Is there anything else that SQLGlot provides us? It supports SQLMesh with, I can understand SQL, so because SQLMesh can understand SQL, there’s a lot of features that it provides. It provides basic support to SQLMesh to translate across different data warehouses. Is there anything else SQLGlot’s really doing for people?
Toby Mao: A couple more things. One, it has more than just parsing. It can also do type annotations and logical planning and all those kind of things, which we take advantage of. With SQLMesh, you might want to understand how all the types flow through your system, and we use SQLGlot to do all the type inference and stuff like that.
James Winegar: That way if you do a join on a varchar and now an integer, it’s smart enough to figure out, hey, you’ve got a typing issue.
Toby Mao: Right now we don’t validate that because there’s so many different coercion rules, and we support many different warehouses. But it’s definitely possible. We have the architecture for it, but we don’t enforce that right now because we don’t want to give false positives. We don’t want to say, Hey, you’re not allowed to do this, but in reality you can. That’s a very frustrating experience.
James Winegar: It works on Snowflake, the example I just gave. But it might not work on some other one. I’ve talked to many customers who have tried these other products that say they do a lot of these type checking things and basically they’re overly strict. It says, Oh, you can’t do this. Oh, you can’t add an int to a date or something like that. But you can. But then they have to rewrite their code. It’s really annoying because now there’s two different sources of truth. One is their compiler locally and then one is the actual warehouse. The warehouse is the actual source of truth for us at the end of the day. Getting that to match exactly with a very small team is a very, very hard task. Well also there’ll be changes and things like that too. What’s the value add to the customer versus the level of effort for y’all.
Toby Mao: That’s exactly right. We try to take a balance where we basically tell you something is wrong when we’re sure of it, but if we’re not sure or there can be any ambiguity, then we’ll let the warehouse decide. Because we have state, it’s not so bad when the warehouse tells you because we don’t have to start all over again. We just start from where you left off.
James Winegar: It’s a five-second penalty instead of a 20-minute one-hour penalty. Exactly. Now your iteration cycle is back to developer experience, your iteration cycle’s tightened up by the state management. What’s next for SQLMesh and Tobiko and is there any exciting stuff that you want to talk about with features?
Toby Mao: On the open source front, we recently launched a really cool multi-gateway feature. Within one project, you can actually pick and choose your engine. For example, you could have a BigQuery main project, but then there are some workloads that you want to use DuckDB for. You can just mix and match. That’s really cool. You can also write that all in BigQuery, but then execute that BigQuery code using DuckDB. It just opens up a whole world of possibilities especially with Tabular’s acquisition and Iceberg and data lakes being more and more popular and mainstream. It makes it so that you can pick and choose the right engine for the right workload. That’s something that I’m personally very excited about.
James Winegar: I was talking with some other people at a data warehouse vendor around how that’s really going to enable a lot of utilization that might not have gone to them because with the ability to choose your execution engine it’s not like your storage and your execution are coupled. First we separate out the storage and the compute, but now we’re separating out not only is it separated architecturally, it’s logical too. Now I can say, Okay, I’m going to use Snowflake on this Iceberg table or I’m going to use Databricks on this Iceberg table or I’m going to use DuckDB on this Iceberg table, because it’s 10 gigs or something. I think it really opens up a lot of things.
Toby Mao: From the business side of things or what we call Tobiko Cloud, we’re making that what I believe is a really great product and an extension of SQLMesh. We fundamentally believe that SQLMesh as an open source product should be a complete and powerful tool. It should be able to work and scale at any company. We wanted our cloud offering to really extend that and make that an even better experience so that if you want to really scalably run an operation and have operational excellence and get a bunch of security features, that’s where you go to Tobiko Cloud. We haven’t quite publicly announced it yet. We should be announcing it probably in Feb or March of this year. Basically we have many of the core aspects of our cloud platform that is really going to make running SQLMesh more efficient, have more security and oversight and be more operationally maintainable. Giving you runtime histories, costs, failures, all the logging, DAG debugging, scheduling, all those kind of things that you would expect out of a robust scheduling platform. We’ll announce that soon.
James Winegar: That’s what we’re gonna get about the cloud product right now. In the future maybe we can talk about it more.
Toby Mao: Sure.
James Winegar: Where can people learn more about Tobiko which is your company?
Toby Mao: We have a growing Slack community, you can join us there at tobikodata.com/slack. Come there, check us out, say hi. We’d love to have you be a part of the community.
James Winegar: Awesome. There’s also the website. Of course the website is tobikodata.com and our open source website is sqlmesh.com. It’s a great tool. Everybody should try it out, especially if you’re using dbt and things like that. It solves a lot of problems for you. All right, thanks Toby for coming. Hopefully we get to do something again later once Tobiko’s released and have a great day.
Toby Mao: Thanks for having me.
Jason: That’s it for this episode of Data BS. If you enjoyed this episode, make sure you subscribe wherever you listen to your podcast to not miss the next one. This episode was sponsored by CorrDyn, a data consultancy that helps organizations empower their data. If you have a data challenge, we can help. Visit corrdyn.com, C-O-R-R-D-Y-N dot com to learn more. See you next time.
