Listen on
For many data leaders, the promise of powerful analytics often comes with a hefty price tag and operational complexity, particularly when dealing with datasets that outgrow local memory but don’t quite demand a full-scale distributed warehouse. This episode cuts through that challenge, revealing how DuckDB redefines what’s possible on single-node systems. It delivers high-performance analytical processing directly where your data resides, reducing reliance on expensive, multi-node cloud infrastructure for common workloads.
CorrDyn host James Winegar talks with Alex Monahan, a Developer Advocate at MotherDuck and a contributor to DuckDB Labs. Monahan offers a direct look at how DuckDB’s unique embedded architecture and latest advancements — including significant performance gains in version 1.2 and intelligent out-of-memory handling — provide a compelling alternative to traditional analytical tools. They also discuss MotherDuck’s cloud platform, which extends DuckDB’s benefits to multi-user environments without sacrificing its core strengths.
The conversation covers DuckDB’s core design principles, its rapid development pace, and its role as a flexible engine for everything from local data science to enterprise ETL. Listeners will gain clear insights into leveraging DuckDB for efficient data processing, its strong open-source foundation, and how it can integrate into existing data architectures to simplify workflows and cut costs.
Key Takeaways
DuckDB extends single-node analytical capacity through intelligent memory and disk management.
DuckDB 1.2 significantly boosts performance for real-world analytical queries by pushing down filters and improving top-N calculations. Critically, its streaming engine processes large datasets in chunks, automatically spilling and compressing intermediate results to disk. This allows a single machine to handle terabytes of data that would typically require distributed systems, making powerful analytics more accessible and cost-effective.
DuckDB offers a flexible, embedded engine for diverse data workflows, from local analytics to cloud ETL.
DuckDB’s design as an in-process library makes it an adaptable tool for data science, embedded business intelligence, and efficient extract-transform-load (ETL) pipelines. It reads and writes various formats, including Parquet, and can directly query external databases like Postgres and (soon) legacy systems via ODBC. This versatility simplifies data movement and processing across disparate data sources without additional middleware.
MotherDuck brings DuckDB’s power to the cloud with enterprise-grade features and dual execution.
MotherDuck transforms DuckDB into a serverless cloud data warehousing platform, addressing its single-user limitation with access control and isolated compute environments. It replaces DuckDB’s local storage engine for cloud object storage while enabling a ‘dual execution’ model that combines local DuckDB performance with cloud scalability. This allows organizations to transition DuckDB workloads to the cloud seamlessly, maintaining ease of use with robust governance.
DuckDB’s open-source model ensures rapid innovation and long-term confidence.
The DuckDB Foundation owns the project’s intellectual property, committing to its MIT license and ensuring its long-term stability. DuckDB Labs drives the project’s rapid development, delivering frequent performance improvements—sometimes 30% per release. This model, combined with an active community and an accessible extension system, provides data leaders a reliable, high-performing foundation for evolving data strategies.
Related: CorrDyn offers deep expertise in data engineering and helps clients with data cost optimization. We also build powerful business intelligence solutions and advise on technology strategy for evolving data architectures.
Full Transcript
Jason: Welcome to Data BS, the show dedicated to tackling the big questions impacting the world of data and ML AI without any of the BS. My name is James Winegar. Each week I sit down with guests from across the data ecosystem to unpack how they’re shaping their businesses or the businesses of others through their real-world application of data engineering, ML AI, infrastructure, analytics, and more. No fluff, unfiltered, but slightly edited to remove noise. This is Data BS. Let’s get into it.
James Winegar: Alex, it’s nice to have you. We talk all the time, so how about you introduce yourself?
Alex Monahan: Hi, I’m Alex. I’m a forward deployed software engineer at MotherDuck. We’re actually renaming that to customer engineer, because it makes a lot more sense. So I’m a customer engineer. I also work part-time for DuckDB Labs. I work with them on mostly blogging, a little bit of documentation, and some DevRel stuff.
James Winegar: Today we’re going to be talking mostly about DuckDB, but you can’t talk about DuckDB without talking a little bit about MotherDuck. How are you going to have your DuckDB Labs hat on but then MotherDuck’s going to pop in every now and then?
Alex Monahan: Yep, got the uniform and everything.
James Winegar: I really like DuckDB, you really like DuckDB. It solves a lot of the problems we run into. But I think it’s probably worth talking about just what is DuckDB, what makes it unique, how does it compare against other database systems or OLAP or OLTP? We might even need to talk about that too.
Alex Monahan: DuckDB is the SQLite for analytics. SQLite is famous for being the world’s most popular database by a large margin. Orders of magnitude more SQLite in the world than any other database, probably all put together. And it’s because it’s an embedded database, which means it runs inside the language that’s calling it. So if you’re running Python code, SQLite is actually just a library. It’s inside of your other language. DuckDB takes the same approach where it’s a database as a library. It runs right inside of whatever other language you’re using. But it’s designed for a different type of use case than SQLite. It’s designed for analytics. And that usually means doing bulk operations. Things like reading your whole table instead of just grabbing a couple rows, joining really big tables together, doing big aggregates on it. Really workloads that traditionally, well, I don’t know if you could say traditionally now, but the last couple years have been done in data frames. DuckDB ends up being a really good fit for doing that in a SQL oriented database instead. Other things about it is that it’s a traditional relational database, so it’s what you’d expect if you’re used to using a data warehouse, it’s all the same syntax there. It follows Postgres syntax very closely, but it happens to be in this tiny super easy-to-use package that you can put anywhere. You can run it on your laptop, you can run it on a big beefy server, you can run it in a lambda function, all over the place. It’s a tiny database that runs anywhere. Of course, it’s designed for analytics, which also means it’s very fast for analytics. That’s another key aspect of it: it really is a cutting-edge tech in an easy-to-use form factor and that’s part of the secret sauce.
James Winegar: One of the things is the concurrency is actually a very similar model to SQLite as well where really one user on a database at a time. And there’s some interesting stuff that MotherDuck has done to manage that a little bit more. But we’ll probably get into some of that potentially later. If not, I did a recording with Tino before for MotherDuck that has a lot of that stuff. One thing that’s important is making sure people understand the way that everything is structured from an organization standpoint. Can you explain the relationship between DuckDB Labs, MotherDuck, and the DuckDB Foundation?
Alex Monahan: Absolutely. DuckDB Labs is a company that’s founded by the creators of DuckDB. Hannes Mühleisen and Mark Raasveldt, as well as numerous other folks. They’re at 15 or 20 folks at DuckDB Labs these days, and they’re based in Amsterdam. They’re a company that’s all around custom development and support for the open source project. They’re the main stewards and shepherds of that open source project. If you’re an enterprise and you want to use DuckDB internally for a data platform, they’re helpful for that. Also if you are a company that’s building it into your product, maybe you’ve got a business intelligence tool or a data workflow tool or an ETL tool, and you want DuckDB as a part of your platform as a service, they can also help with custom integrations or faster bug fixes, things like that. They’re supporting that open source project. The open source project intellectual property is owned by a nonprofit foundation, the DuckDB Foundation. What’s nice about that is it has a strong shepherd in DuckDB Labs where there’s a whole set of folks that are making it better and better all the time. It’s not owned by that company, which means that it can’t be bought by another company, it can’t really be restricted in any way. It’s an MIT licensed open source project and it will remain that way forever, because it’s legally obligated to remain MIT licensed, which is really nice, gives you a lot of long-term confidence in a database, because when you choose a database, you’re choosing it for a long time. You want to have long-term confidence there. MotherDuck in contrast is a very close partner with DuckDB Labs, but it’s a venture-backed startup based in the US and we’re globally distributed at this point. We’ve got around 50 folks and we are taking DuckDB into the cloud and building a serverless cloud platform with DuckDB as the engine. We’ve got a couple different pieces of secret sauce. A large part of it is that we are huge fans of DuckDB and even the core competency of being a DuckDB service in the cloud, we think is a big value already. We also have some other secret sauce around, we’ve replaced how DuckDB does storage. Typically DuckDB does storage on a single file, which is great on your laptop, it’s not great in object storage because you’d have to replace the whole file even if you inserted a single row, which would not be fun if you’ve got a couple terabytes. We’ve replaced the whole storage engine and built a cloud service around it. We’ve got some other secret sauce around running it combined with local DuckDB. Lots of fun additional things on the MotherDuck side. In terms of the relationship, DuckDB Labs and the founders there own a portion of MotherDuck. They’re co-founders alongside the founding team of MotherDuck, which means that they succeed as MotherDuck succeeds as well. The folks that created this amazing open source project also get to reap the benefits of the cloud service, which is really what you want to see in open source. In addition, there’s a contracting relationship where DuckDB Labs does a lot of things to help MotherDuck. If there’s a new hook that needs to be added or an area of the codebase that MotherDuck is taking more advantage of than the open source side is, they can help prioritize things for us.
James Winegar: I’m nearly positive that funds several engineers on the DuckDB Labs side to be focused on extending as appropriate to support the data warehouse, DuckDB service in the cloud model that MotherDuck has. Definitely helps. In many ways they’re really the fundamental experts on DuckDB itself. They’re really great partners. Now I think we got a good understanding of DuckDB, MotherDuck, the foundation, etc. Today’s goal’s really focused on DuckDB. We’re going to be mostly talking about the 1.2 release that’s coming out, I think, next week. By the time this recording goes out, it might be a couple days before or a couple weeks before, we’ll see. What are the major highlights of DuckDB 1.2 to you?
Alex Monahan: DuckDB 1.2, I like to think of the top line highlight as performance improvements in the real world. DuckDB has really all been about combining research and practicality from the very beginning. It was not designed to be a research prototype, it’s designed to be a database for production powered by research. What that has really meant is that they’ve built a solid foundation and a usable system first, and then they’re adding incrementally more and more speed over time, because they already have that really efficient core. This is one of those cases where they’re adding a lot of speed in a lot of really useful cases. A lot of times where you’re doing queries where you’re looking at a list of things or doing filters with OR, those are going to now be pushed down into the storage and those get far more efficient. A lot of real-world queries will just get a lot faster with 1.2. There’s also some big benefits with the Top-N type of calculations, like what are my top 10 customers? Those get quite a lot faster. And if you’re dealing with long strings, which database benchmarks famously ignore, a lot of the research around database systems they think everything’s an integer, everything’s a float, we don’t need to worry about strings. Well, DuckDB is now compressing strings while writing to the database, when you’re storing it on disk. So you can have long strings and it will be a nicer performance there as well.
James Winegar: Sorry, that’s actually a big deal for me. Does that have impact on the query performance at read time?
Alex Monahan: It’s really a benefit all around. In general, even on a laptop, our hard drives, our SSDs are a lot faster than they used to be, but they’re still slower than CPUs. And that gap tends to be widening over time. CPUs get fast and it’s hard to get that storage bottleneck. It’s a great trade-off to compress and store and then retrieve from storage and decompress. The reads and writes are faster and you save disk space.
James Winegar: Overall throughput is higher basically with it on the read and the write side. Okay, cool. Sorry, that’s actually a big deal for the stuff we’re — I have a use case where we’re using MotherDuck and DuckDB, and me and Alex have worked together a lot on some of these things that are actually coming out in 1.2 and 1.1 we had some pretty major things we were working with DuckDB Labs on to try and figure out why we were running into problems. The team’s been very responsive to: hey, this is a real-world use case, this is a use case that is reasonable. We should make sure that we can handle these pretty decent sized datasets and how do we effectively go through the processing. We had some — it’s pretty complicated. Out of memory type situation. You have to be smart about how the query engine’s executing all of that.
Alex Monahan: You bet. I think this is a very common workload, especially if you’re working with JSON data. JSON data is stored as strings in DuckDB. If you’ve got event data, if you’ve got logs, anything that’s JSON format, gets a real big boost from this.
James Winegar: That’s going to be nice. I’m actually really excited for the customer database that I’m thinking about. Let’s just transition into out-of-memory datasets and, with particular to 1.2, how is DuckDB handling my table, let’s just assume I read the whole thing and it’s 400 gigs but the device it’s running on has 36 gigs or something like that. How is 1.2 going to change the performance in that situation?
Alex Monahan: In general, DuckDB’s got a couple ways that it handles that kind of large data set. It is, I think, a real good differentiator for it. A lot of the data frame tools out there and other things that people use in that ad hoc world for DuckDB, this is a really tough thing. It takes a lot of effort to build. And DuckDB invested a year or so or more into building this capability. One is that the engine itself is designed to be a streaming engine. It’s not a streaming database, but it streams off of disk, which means it reads the data in chunks instead of all at once. You never have to read all 400 gigs into memory at once, you can process it one chunk at a time. With DuckDB, it actually breaks into pretty small chunks, so you can keep your memory use pretty low in general. It’s chunks of a hundred thousand rows. It’s really just pulling exactly what it needs off of disk. The second part is that if it ever has more pulled off of disk, or it’s doing an intermediate calculation that’s larger than it needs, it can buffer data back to disk. And say, I’ve been keeping this hash table in memory but it’s too big, or I have too many of them. Let’s put that on disk for now, and in a later stage I’ll read it back off of disk and aggregate it back into my final result hash table. That happens in joins and in group bys, so it’s common. Window functions have separate structures, but they also have out-of-memory support as well. Those intermediate calculations that get spilled, when they get spilled to disk in 1.2, they’ll be compressed first. That’ll be really nice to handle larger and larger structures that can be spilled to disk.
James Winegar: Which is exactly our use case. We have high-cardinality group bys across five different columns plus-minus. We have these huge hash maps that get built out to support the thing. Anything that improves those processes is fantastic. I’m excited.
Alex Monahan: It expands the top end of what you can do on a given piece of hardware with DuckDB. If you were limited by your hard drive space, well now you can fit two, three X more on your hard drive while you’re spilling. The total size of problem that DuckDB can handle is taking a nice leap with 1.2.
James Winegar: Which also means that the resources that are available to deploy easily in a cloud provider like AWS, I can start to handle a bigger and bigger workload. With being able to handle that, it’s going to allow DuckDB to be very competitive with a lot of the bigger players as far as performance, except for scanning a petabyte of data. Okay, well, how many actually do that? It’s a whole discussion there. There’s 0.001% of the queries are actually hitting the tens of terabyte, petabyte range, if you even have that much.
Alex Monahan: Exactly. We think there’s a lot of cases out there. You could really take that benefit in two ways. You can have a really big beefy machine and now it can do more and more, you can handle really giant problems with DuckDB. Or you can run on an even smaller machine than before with DuckDB at even lower cost. You can pick which benefit you want there. I think that’s also core to some of the strategy of DuckDB, which is take a single node as far as it can go and use the disk that’s already there to its fullest extent, because we believe that the algorithms of dealing with disk can actually be far simpler and more efficient than algorithms dealing with many multiple nodes over the network. It also ties back to the pace of improvement of DuckDB. If you’ve heard about DuckDB a couple years ago and looked at it then, you should look again. It is improving at a crazy rate, especially for a database. They’re not simple pieces of code. It’s a million lines of code or something like that in DuckDB. It compiles down to 20 megabytes, but there’s a lot there. Because there’s a smaller problem space, the improvement rate, sometimes it’s 30% improvement per release. And that’s a couple times a year. That’s wild.
James Winegar: I’m guessing a lot of the work that the team has to do is in how many threads do I need for this process based on a bunch of heuristic information and things like that. I don’t get into the internals of DuckDB. But I’m positive that some portion of it is: this portion’s going to be able to run, scaled out pretty easily based off of how much data we’re talking about. I got eight row groups, so I’m going to use eight threads or something like that.
Alex Monahan: It’s a pretty dynamic way that it breaks things apart. It’s really grabbing pieces of work and knocking them out as fast as possible and having all the cores be able to go ahead and grab it. DuckDB is very good at using all of your cores at once. It’s a multi-threaded model as most modern databases, Postgres is the big outlier being process-based instead of thread-based. But in general being thread-based allows you to really split the work out across the cores very well. DuckDB uses morsel-driven parallelism, which is a very dynamic way of on the fly just grabbing chunks and working on them. Some other types you actually have to do that in the planning stage and you have to come up with a plan for 190-some cores before you even start running. Those plans are huge and complicated. But DuckDB allows it to be very dynamic and just as cores are available to work on more they grab it.
James Winegar: Streaming the data in is also important when you get into that because that’s another detail you don’t have to quite worry about. Unless you got 256 cores or threads or whatever, you got 20 gigs of memory, well, you’ll run into issues. But if you have what you would expect from the memory footprint for something like that, you should be in a good spot. What types of problems do you feel like DuckDB solves very well? And which ones do you think it doesn’t solve well, based on where it’s currently at?
Alex Monahan: I think the goal is for DuckDB to be a Swiss Army knife. There’s a wide variety. In general, it is aimed at those analytical workloads. I think the original use case for DuckDB was data science. If you’re running a Jupyter Notebook, you’re running an R script, and you’re already working in that other language, how could DuckDB meet you where you are and be a database library that you could use in place of a data frame library to bring their cutting-edge database research into the data science realm. That’s a core use case today still. It’s a huge part. The other part is it’s a really great ingredient for use cases. If you are building a tool or platform, tends to be a really great component of building a tool. There’s a lot of creativity that can go into that. There’s business intelligence tools that use DuckDB, because anytime you change a chart, I want to now group by and split out one line per company. That’s a data operation and it’s an analytical one because it’s usually a bunch of data you’re working with. DuckDB’s a really great engine for charting. It’s also great for ETL type of pipeline workloads. DuckDB can read and write Parquet, it can read lakehouse formats as well. That means it’s very good for moving data around in the cloud. It can write to all three object stores in the cloud, all the different cloud providers, read from them as well. You can pull a Parquet file from one cloud, compress it down, reformat it, do any SQL processing on it you want and rewrite it somewhere else. It’s a great tool for doing this kind of ETL or ELT style flows. Also, it does have some cool query federation capabilities. You can query a Postgres database directly in DuckDB and you can then also query SQLite databases, MySQL databases, stuff like that. It’s also got some of that query federation style workload, data virtualization is the other buzzword for that that’s been around for a bit. But you could do data virtualization as well. Those are cases where I think it’s really powerful. I think that at MotherDuck we’re pushing the envelope of it in the core data warehousing use case. DuckDB as an engine is a pretty good fit for that, because it’s amazing at joins and group by and sorting. It has world-class algorithms in all those categories. Some other databases out there are really good at aggregation but not so good at joins or vice versa. DuckDB is actually very well-rounded, so it’s good for generic data warehousing as well. The areas where an analytical database in general has a harder time is if you want to be inserting or selecting really small amounts of rows with a ton of concurrency. You hit on this earlier where it’s an in-process database, which means it’s really more of a personal database or an individual database and not a used by everybody database.
James Winegar: A big one of the ones very DuckDB specific that I think is interesting is DuckDB inside of Postgres. We’re excited about that. It’s offload the stuff that Postgres isn’t good with to DuckDB, because DuckDB speaks Postgres protocol, storage protocol too. It can read write directly on the same file structure that Postgres is managing. Now basic stuff, you can really do Postgres for your database, until you scale and or until you get high volume analytical workloads because you don’t want to run those analytical workloads on your production transactional database because you can run into issues of resource congestion because DuckDB will consume the resources. You don’t want to get into a capacity management thing. But there’s a lot you can do from that, maybe read replica, DuckDB on the read replica, that’ll take you a decent distance. Then you got to look into something else like a proper data warehouse implementation because you need multiple users, integration with a bunch of other things. That’s where MotherDuck comes into play, but you can have a really seamless transition as you evolve through that life cycle of building out a product and things like that. That’s one of the ones I’m very interested in, conceptually anyways.
Alex Monahan: I would say it slightly differently just that Postgres is also having a moment. DuckDB if you look at the popularity graphs, it’s shooting up probably faster than anybody right now. But Postgres is also still shooting upwards and it’s been shooting upwards for decades. It’s really having a moment by the last five years or so where people have aligned that open source has reached the level of some of the enterprise databases. MySQL was the other open source darling database, Oracle bought it, there’s Oracle, there’s a lot of people who are trying to move to Postgres from Oracle because it’s got a lot of the same fundamental viewpoints. You got your SQL Server, if you’re not doing C# type stuff, the integration isn’t there to support you as much. Plus licensing, etc. Virtualization, licensing has cost people a lot of pain and frustration during audits that they get involved in. Having an open source database where you’re just like I run it on hardware and it’s battle hardened pretty well at this point. Your community’s very good about making sure that no nonsense gets in, so it just continues to get more effective and more effective over time. What does that mean? It means that these proprietary solutions aren’t as valuable as they used to be. Used to be that for a lot of things you had to run an Oracle database. There wasn’t another option, your scale was too big, there’s nothing else that would have enough juice or your complexity of your use case was too complex, nothing else had enough juice. I’ve worked in Oracle land, not so sad to be not writing Oracle queries at the moment. Postgres is also another similar approach to DuckDB in that it built a solid foundation of extensibility and reliability and transactionality and they’re adding the speed on over time. Postgres is getting faster with each release. MySQL is actually getting slower. Every release MySQL gets a little slower, which is wild to think about. Spoiled in DuckDB land, stuff just gets so fast all the time. DuckDB has a couple cool ways to tie into Postgres. You can use the Postgres scanner, so you can read and write to Postgres as if you were talking to Postgres through any other client connection, which is very powerful and it can be transactionally safe. You can actually read and write to Postgres within transactions in a whole separate database. Pretty wild. But even more wild is that there’s now an extension for Postgres called pg_duckdb. This actually embeds Postgres — or sorry, embeds DuckDB in a Postgres extension and that starts to really open up the seamlessness where you can just connect to a database like it’s Postgres and DuckDB takes over the execution. It’s really a new level of seamless. It’s in beta, try it out.
James Winegar: What are you really looking forward to coming out of DuckDB this year, as we look towards 1.3 and things like that?
Alex Monahan: I think there’s a lot of exciting things. In general, the DuckDB roadmap is by design pretty fluid. There’s a lot of opportunities where as you have a use case where you’re stretching DuckDB, reach out, talk to the DuckDB folks on either Discord or on a GitHub discussion. I think there’s a lot of cool directions to go in. Some of the ones I’m excited about are there’s plans to add an ODBC scanner, much like the Postgres scanner we were talking about. Essentially, you could from DuckDB connect to DuckDB. You could then reach out and query a wide variety of other databases like Oracle, like SQL Server, like IBM DB2 through their ODBC driver. Suddenly you can easily pull data out of those legacy systems and into a system that’s much better for analytics and do it all in SQL without a full separate ETL flow with some separate tool. You could actually have a SQL statement, Select Star from IBM DB2, insert into DuckDB. Or insert into Parquet file on S3. Really pretty cool workflow engine for moving from legacy or more transactionally oriented databases into the analytical world of DuckDB or Parquet on an object store, data lake land.
James Winegar: I was just thinking about how you also have to be careful when you use an ODBC scanner against those other systems for the same reason we were talking about with Postgres. Get resource contention. You do the select star from a production database. Message. Make sure you try and do an incremental load or try to, and the caveats before that. Only grab the stuff you need. It’s row based so it’s not as big of a deal on the star side, but don’t grab the whole table, you’re going to blow out the buffer cache on those systems. Be thoughtful about the approach. But it really means that you can do the fast-paced analytics that you couldn’t really run on Oracle directly for whatever reason, because when it does the aggregation it’s very expensive because it’s a row-based system. There’s things you can do there, but we bring it over to DuckDB, it’s for that purpose. As long as you’re not getting some craziness over the line, you got 10 gig thing that you pull over, it’s going to take a few seconds and you’ll be done. Then you do what you need to do from that. If you got look at the last 30 minutes of data, is there anything I need to take an action on? Yes, no, boom. Those types of queries, I think, you can offload off of the Oracle execution engine, run it from a server that’s next door to it, and really stretch out your Oracle database more effectively or your SQL Server or whatever.
Alex Monahan: You bet. Absolutely. I think one of the patterns that I really like for that is have a periodic job to grab data in bulk, hopefully from a replica. But then if you really need up-to-date data, combine the bulk data that’s already in DuckDB with the latest and greatest from the transactional system. Use the transactional system for what it’s good for, just get me a tiny bit of exactly what I want and combine it with your historical data that’s in efficient DuckDB format. I think that’s a cool pattern.
James Winegar: We’ll take it one step further. Get the latest thing from the DuckDB side and then when you go back out to the Oracle side or whatever, now you have your watermark to move across. Pass the filter. Now you don’t have to do deduplication or anything else reasonably if you have a field you can use to distinguish that like an updated timestamp or something.
Alex Monahan: You bet. Absolutely. I think it’s very interesting because it’s again the real world, right? The real world is these databases have been around and they will be forever. That’s one. Another is really around data lakehouse formats. There’s a lot of work going on in the data lakehouse space around catalogs, around Iceberg, around Delta, and DuckDB is heavily involved there with all of that. What folks are finding if you’re building a catalog and you want people to be able to use it, easiest way is to build support for DuckDB. Rather than having to run another service to go operate your catalog. DuckDB I think is actually going to be a really great way for catalogs to drive adoption because it’s so easy to use. Lots of exciting stuff in the catalog space.
James Winegar: I don’t understand the connection there off the top of my head, so what’s the primary driver for why supporting DuckDB helps manage the catalog more effectively?
Alex Monahan: If DuckDB can read your catalog, it just opens it up to be much easier than if you had to run something like an AWS Glue to go work with a catalog.
James Winegar: Oh, okay. Because DuckDB is in-process, you can run it wherever you need to run it and then if your catalog supports DuckDB then you can make your deployment strategy not as complicated.
Alex Monahan: Yes. The catalog would still be managed somewhere separately and operated, but it’s much easier to connect to and consume from that catalog if it’s DuckDB compatible.
James Winegar: This kind of ties into I think what Databricks was doing with DuckDB: Unity Catalog working with DuckDB to then expose basically everything from there.
Alex Monahan: There’s a good partnership there for sure. That’s part of the benefit: the DuckDB ease-of-use and accessibility with the ability for a catalog to work for a whole organization. Awesome. Lots of fun stuff there. I think there’s continued to be more and more other fun stuff with DuckDB. Lots of other goodies there, but I think those are a couple for sure. Another area that we’re really excited about as well are community extensions for DuckDB. Extensions have been a part of DuckDB for a little while. Some of the core capabilities that folks love about DuckDB like the Parquet reader and writer have been an extension for quite a while. But more recently, these extensions can now be built by anybody in the community. It’s a lot simpler than before because there’s a template on GitHub that you can clone, and it comes with a CI/CD workflow that will compile everything and test it on all the various platforms. You can even then, once you’ve got it all tested in your own repo, commit just a description and a link into another repo, and DuckDB will actually do all the distribution as well. They’ll publish it for all the releases that you have and people can actually treat it a lot like Python’s PyPI, the Python Package Index, and they can actually just say in any DuckDB, it’ll automatically download the binary that you need and you’re up and running in a couple of seconds.
James Winegar: Is there a configuration option to turn that off?
Alex Monahan: You bet. Yeah, there is. In all these things that we’ve talked about, DuckDB is by default designed to be very open. It’s a Swiss Army knife. The goal is to let you pull data from wherever you’d like. If you are building it into a system where you do want it to be an engine that does a specific task, there are lots of ways to lock things down. I think at least one of those was a request that I made back at my Intel days: could you let us turn that off please? They cheerfully applied. It was really helpful for us. But yes, you can use that only if you really want. But that’s really exciting. DuckDB is in my opinion, a little biased, the world’s best single-player or single-use database. It’s amazing. Wherever you can place it, it’s going to turbocharge that workflow, whether it’s you on a laptop or you’re building an incredibly powerful giant ETL system. It’s the best ingredient. However, DuckDB is also pretty cool in that there’s a page on DuckDB.org, it’s Why DuckDB, and it includes cases for why not DuckDB. It explicitly lists data warehousing as one of the cases where DuckDB is not built for that. It’s because of its single-process nature. What we’ve done at MotherDuck is taken the amazing parts of DuckDB and we’ve extended it so that it’s also an amazing choice for data warehousing. By having it be a cloud platform, you get the lovely syntax benefits of DuckDB, it’s the easiest friendliest SQL to write. You get that fast execution, that scalability, spilling to disk, all that type of stuff. But we’re also trying to add a lot of value in the service layer. Common most of the time you see a cloud platform on an open source project, it’s fairly similar, they’re going to run it for you. But in this case, there’s a lot of components that MotherDuck has added. DuckDB has no concept of a user, because you’re in-process. Which means there’s no permissioning. If you’ve been a DBA or you’ve administered a cloud data warehouse, you don’t really want to share everything with everybody. Generally you want to have some security. MotherDuck, for example, allows access control. It also allows you to do things like have each person using your database have their own compute environment, so that way you can allow them to work totally independently and not step on each other’s toes. There’s a lot of benefits to the service architecture that MotherDuck has there. Lot of fun there.
James Winegar: The serverless approach to data warehousing with this I think is another valuable piece.
Alex Monahan: The serverless approach I think is definitely huge as well. Part of our mantra is to be easy data over at MotherDuck and serverless just makes it very easy to spin up just as you need it to only run the compute workloads that you need, and by having it be by individual, it really allows for nice workload segmentation and allows for just a lot of nice benefits that way. That serverless aspect is a huge win as well. The last part of our secret sauce is what we call dual execution. We used to call hybrid execution, now we call it dual execution. There’s a lot of other things that use the term hybrid. For us, that means when you connect to MotherDuck, you use the DuckDB driver you already have. If you’re in Python, you’ve already pip installed DuckDB, or if you’re using ODBC, it’s already ODBC driver, and that will actually form a bridge between your driver and MotherDuck. But what’s unique about DuckDB is that driver, that ODBC driver, it has the whole database engine in it. Wherever you’re running, you have an engine in the cloud, a MotherDuck, and an engine locally. That could be locally in your GitHub action, somewhere in the cloud, in your lambda, or on your laptop.
James Winegar: Pick wherever you want it basically. Alex, what’s it like working in MotherDuck and what roles are currently open there?
Alex Monahan: It’s a whole lot of fun. Where I worked previously was a giant company, tons of great people, really enjoyed my time there, but working at MotherDuck is great for the community aspect to get to really know everybody, I can just get help from everybody and it’s a very collaborative environment which is really exciting. There’s a ton of variety, because we’re moving fast. DuckDB moves fast, MotherDuck moves fast, we’re building a lot of fun things. For me in particular, I’m a fan of the technology, I’m a fan of the approach, I’m a fan of the vision, and it’s so easy to work somewhere where you’re a fan of it. I feel so lucky and fortunate both on the MotherDuck side and the DuckDB Labs side. The thing I’m a fan of I get to work on every day. It’s just so great. I know it helps folks. It’s legitimately more efficient, or cost-effective, faster than what they’d be doing otherwise. It’s easy that way. In terms of open positions, we’ve got some on the software engineering side, in the platform side and a couple other software engineering roles there. We’re also recruiting on the product side, bringing in some expertise in product management specific to databases. A couple other spots. Take a look at MotherDuck.com. We’ve got a careers page. It’s a really fun place. One of our values as a company is unorthodoxy. If that’s you, come on down. I think that adds to our culture of being really collaborative: the best way to work together is to accept viewpoints from all over the place, and unorthodoxy is I think a fun part of that.
James Winegar: I really enjoy working with everybody on the MotherDuck team. We talk a couple times a week. It’s a pretty connected environment. Back to DuckDB, my bad. How can people get involved in contributing to DuckDB? We talked GitHub, we talked Discord. Is there anywhere else that makes sense?
Alex Monahan: Those are great. The community extensions are also a really great way to jump in. I think of Python, in many ways it has been an amazing wrapper language. People building AI and ML models aren’t writing C++, they’re writing Python that calls C++. DuckDB could be the same where you can write SQL to call into C++. You’ve got any kind of cool C++ library that you want to wrap, DuckDB allows it to work in any language, Python, R, JavaScript, all over.
James Winegar: Oh, that’s interesting, it’s like the foreign data wrapper but down into a C++ library instead of a function call etc.
Alex Monahan: Kind of like that. It’s kind of like how WASM can be this universal compile target where you can use it all over, same underlying library. Same way in DuckDB extensions. I think that’s a really cool area where folks can jump in. I am not a C++ programmer, but I’ve been able to do a little bit to help with some extension stuff. Even if you’re not a C++ person, the CI/CD, the compiling part, the hard part just works.
James Winegar: The hard part’s the build. If you’ve never set up a build engine for C++ it’s very complicated. But since they already do all that work for you, all you have to do is write some code. We live in a world where basic stuff you can get a lot of help with from ChatGPT and things. As long as you’re not doing crazy or state-of-the-art, you can actually learn a lot and get community feedback by contributing to an extension. If you’re wanting to learn and you got a cool idea, maybe that’s a good idea too. What would you say are the key takeaways for people listening about DuckDB?
Alex Monahan: I think with DuckDB, try it out. It’s MIT licensed, it’s free. It is incredibly popular for a lot of good reasons. It’s designed to be fast but more importantly easy. There’s a lot of databases that are the other way. More importantly fast, easy where they can. But DuckDB is easy first and you feel it. It’s really just a joy, I would say, to work with DuckDB, especially in contrast to the databases of yore, the ones in the past. It’s fast and easy.
James Winegar: Good example there is the CSV reader, reading CSVs in DuckDB is a joke as compared to basically anything else. It tries to do a good job of determining all the details, that’s a great example. Go look it up. If you work with CSV files and you hate it, go look at DuckDB, it’ll make your life easier. Where can people stay updated on DuckDB’s latest news and features and community development? Is that Discord, GitHub?
Alex Monahan: Discord, GitHub are great. There’s also a MotherDuck newsletter that also covers the full DuckDB ecosystem. That’s another great way to subscribe and every month or so get a real great download of what’s been happening in the industry. There’s also, you can follow on Twitter, BlueSky, those type of places.
James Winegar: Awesome. Alright Alex, it was great to have you. Hope you have a great day.
Alex Monahan: You bet. Cheers folks, happy analyzing.
Jason: That’s it for this episode of Data BS. If you enjoyed this episode, make sure you subscribe wherever you listen to your podcast to not miss the next one. This episode was sponsored by CorrDyn, a data consultancy that helps organizations unlock the power of their data. If you have a data challenge, we can help. Visit corrdyn.com, C-O-R-R-D-Y-N.com to learn more. See you next time.
