Listen on
Uncontrolled data proliferation and a lack of clear lineage pose significant risks to compliance, data quality, and business decision-making. As organizations blend internal and external data, tracing the origin and transformation of every data point becomes critical—not just for auditing, but to prevent the accidental exposure of sensitive PII or the introduction of biases into crucial models.
Data leaders also face strategic architectural decisions regarding open source flexibility versus proprietary vendor ecosystems, choices that profoundly impact long-term costs and agility. Navigating complex migrations from legacy systems further complicates this landscape, demanding a structured approach to minimize disruption and maximize value.
In this episode, James Winegar speaks with Kevin Hartman, VP of Analytics at Databricks and a data science lecturer at UC Berkeley. Hartman brings a dual perspective, combining real-world platform leadership with academic insight as they discuss establishing robust data lineage, the strategic benefits of open standards, and a phased approach to migrating from legacy systems like Netezza, all while emphasizing the power of community in data science innovation.
Key Takeaways
Granular, automated data lineage prevents compliance risks and hidden biases.
Many organizations unknowingly ingest PII or biased data through complex transformations, especially when integrating third-party datasets. Platforms like Databricks Unity Catalog offer automatic, column-level lineage tracking across languages and queries, linking data changes directly to the originating code. This visibility is essential for identifying sensitive information early and ensuring ethical data use before it impacts models or business operations.
Prioritizing open standards lowers costs and improves talent acquisition.
Proprietary data systems often create vendor lock-in, tying compute and data access to a single provider. Adopting open standards, like the Delta format or Unity Catalog, ensures data ownership and portability, allowing organizations to choose best-in-class tooling. This strategy simplifies talent recruitment by tapping into a larger community of practitioners and avoids costly, custom integrations.
De-risk legacy data migrations with phased pilots and automated tooling.
Migrating from legacy data warehouses like Netezza requires a structured approach to minimize downtime and ensure data integrity. A phased strategy begins with discovery and a production pilot to test the migration plan in a controlled environment. Automated code conversion tools can accelerate up to 80% of the work, while parallel testing and data reconciliation ensure business rules and data quality are preserved.
Active community engagement is vital for responsible data innovation.
The rapid evolution of data science and AI demands continuous learning and shared problem-solving. Engaging with broad communities—across industry and academia—helps practitioners test assumptions, identify emerging ethical considerations, and collectively drive innovation. This collaborative mindset prevents organizational silos and cultivates a culture of responsible, forward-thinking data practice.
Related: CorrDyn helps clients build robust data engineering foundations and establish strong data reliability to ensure data quality and compliance. Our technology strategy services guide organizations through complex tooling decisions, including open-source adoption, and we specialize in managing large-scale digital transformation initiatives, including legacy system migrations.
Full Transcript
Jason: Hi everyone, this is Jason, producer of Data BS. Welcome to part two on our interview with Kevin Hartman, head of Partner Solution Architects in the Americas at Databricks and lecturer for data science at UC Berkeley. If you haven’t listened to part one of this episode, we highly recommend that you go and check it out. We will drop a link to part one in the description of this episode. If you did listen to it, we hope that you learned a lot from the conversation so far and enjoy the remainder of this interview with James and Kevin as they finish their discussion on collaboration, community, and best practices in data science. Here we go.
James Winegar: Welcome to Data BS, the show dedicated to tackling the big questions impacting the world of data and ML AI without any of the BS. My name is James Winegar. Each week, I sit down with guests from across the data ecosystem to unpack how they’re shaping their businesses or the businesses of others through their real-world application of data engineering, ML AI, infrastructure, analytics, and more. No fluff, unfiltered, but slightly edited for room noise, this is Data BS. Let’s get into it. We talked about these clean rooms where you might be bringing in data from various parties. Let’s just use Experian plus Nielsen plus your own data as a working example for that. How can Databricks support the lineage tracking of that data through a set of transformations into a final artifact so that you can trace back, ‘Oh, I actually didn’t use Nielsen or I actually didn’t use Experian’ or ‘I was using Nielsen for these identifiers and I was using Experian for these identifiers’? So now I’m able to have a much better understanding of where we’re sourcing different pieces of information.
Kevin Hartman: That’s another great question about lineage. If you’re going to do a production system, clean rooms is one example of getting access to data in a place that is a shared environment. What is probably more typical is that once I understand what I need, I’ll probably use data that’s on the marketplace outside of a clean room to productionize that. Now that I’m creating a production-level asset or production-level product that is leveraging this data, I’ll be doing my own transformations on my data to conform to that or conform to however I’m going to be doing my analytics, weaving in this other data from other sources. The good news is that if I’m doing work there and I’m changing dimensions or I’m changing the transformations or things like that, if you’re using Unity Catalog to do all that work, all your lineage is automatic. Everything, across queries, across languages, all the way down to the column level. It also includes other attached things about that transformation, like anything that was a notebook or a job or anything related to running that query, that’s also available in your lineage view to see, okay, not only did that thing change, but here’s the actual code that was used when that change was made. That’s really important to understand how I can track all this information as things might change. The challenge is what would occur with these external systems where you don’t have that tracking mechanism in place, it’s not automatic. Either you accept that I’m not going to be able to do that, but rely on that source system, or if it’s a source system that also reports on lineage information, the good news is that you can begin to ingest that as well and track how that might have changed. It just takes additional work. There’s an API that allows you to import and ingest the lineage information from the source systems as well. It just takes additional work. You need to understand what those APIs look like and the metadata that’s needed and how to conform what needs to be fed into the API to track it. But then you can have end-to-end view of every single change in your ecosystem.
James Winegar: Yeah, for federated data sets and things like that where you’re actually calling a view definition and that view is really a few tables joined together with some filtering pre-applied to it. And then you want to have context of that in the upstream or the downstream side, so that you actually ‘Oh, I’m not actually calling what the transactional lines are, I’m calling this other thing.’ But if that’s not exposed to you, really the only people who know are the developers who work on that particular app.
Kevin Hartman: Exactly. And if it’s outside your organization then you need to go to that organization and say, ‘Hey, can you please provide me some of your change information that I need to get access to.’ It may not be a problem you need to solve for, honestly, usually. It just depends on the problem statement. It really depends. Usually outside sources keep a fairly unified data description of what they’re providing to you and if they are going to change that, usually there’s just a version for that, that you would say, ‘Oh no, I just need the old version of that, because that’s the data I need, that’s the contract.’ An example situation I can imagine is you don’t realize you have some PII coming over because whatever, it’s got a weird name and you don’t know the name. You live in the US and it’s the Japanese ID number. I don’t know what that’s called off the top of my head, but I know that that’s a piece of PII. And if you’re doing some finance-related thing and that type of information comes into your model, which depending on the use case, isn’t really allowed to have, and then out of nowhere you have bias that would technically be illegal introduced to your model. So you have to go all the way to the very raw data and make sure that you’re not bringing in any of these protected class information where you can’t bring it into the model. Also in the platform it’s part of the data intelligence engine. It’s detecting and looking and doing some observability. There are actually built-in toolsets to automatically detect PII. And maybe not in the case you described, the nuances between a Japanese equivalent and an ID, but sensitive information can almost become auto-detected, at least in a sense, then alerted to you to do something with that, whether or not to go back and protect it or not.
James Winegar: I actually only happened to know about this Japanese ID number because of some data loss prevention tooling and so I know that that’s on the list somewhere. [laughs] One of the things you were talking about earlier is that because Unity Catalog is open source, you talked about these open API specifications for Unity Catalog for you to ingest various metadata and information such as lineage. I think that segues pretty well into choice of open source tooling. Databricks is at a position where they can just build their open source tooling and use that as part of their solution. But if you’re advising somebody, maybe not even within the scope of Databricks, how do you help them evaluate and decide which open source frameworks they should integrate into their stack?
Kevin Hartman: There’s a couple things there around open source and yes, part of Databricks’s philosophy, and it’s core to our belief system, is to make sure that you rely on open standards. That’s our first-class, first-level position. I can go in a couple of different directions with this, but let me answer your question first about choice of open standards and what open source systems. Even before joining Databricks, I’ve been a practitioner and my own philosophy has always been I want to leverage things that are going to be driven by the community or embraced by the community. Those are the things you’re going to look for when you’re choosing open source tooling. How big of an ecosystem is it? Is it actively contributed? When’s the last pull request and commit? If it’s active, and you’ve got a lot of people who are contributing and it’s evolving, that’s a pretty good candidate for selection as to what tool I might want to use because I now understand that it’s not just me who is going to be developing and using this, but it’s also a whole community of people who will. And it also helps my organization. Because if we’re looking for assistance and help other practitioners to develop something, it’s much harder to find someone off the street that might know information around a proprietary system than it is to look for people who have been involved in that community at large and know about the open source or open standards that are inherent and know how to develop towards them. That’s a really important consideration. But then of course you need to consider other things about the tooling, is this solution going to scale for my needs? What is it that I’m going to be using this tool for? Those are other important considerations to factor into your selection. Going back into where my mind was going when you asked the question around open source and Databricks’ philosophy and why we have that position, it’s also because we embrace the community at large for your choice of tooling and the way we think about this is we don’t think you should get locked in to proprietary solutions. That’s why things like the Lakehouse advent came to fruition, because it was community-inspired, community-led, built on open standards. Now it’s been adopted, but we still believe that you should continue that journey and leverage things like simple storage from your data lakes and put on top of that now a governance system which is Unity Catalog, but before that was just a Delta format, an open standard protocol that allowed you to treat your data lakes in a way that supports ACID transactions. Then you’ve got the key ingredients to support what a data warehouse would support, but built on open standards so that I can do all the same things outside of a proprietary approach so that the community at large can support that. Tying those things together, our belief system is that open standards is really high in that list and then providing services around those open standards is where we position the tooling and the product. Don’t lock someone in, but use the tools of choice for the best tools that you have out there, agreeing upon an open standard, an open format.
James Winegar: A major benefit of Databricks is you can do it all in one spot. That’s really what it boils down to for me, that I don’t have to bring in another tool. I don’t have to think about it. I just do the thing within Databricks and I move on with my life.
Kevin Hartman: Yes, you can. But you have choices too. You’re using Databricks because it provides the most fast and efficient way for you to get your workloads done. But you still own your data.
James Winegar: Yeah, you’re writing the data into Delta format at the end of the day. And Delta format, I want to query a table with DuckDB or something.
Kevin Hartman: You can do that.
James Winegar: In a lambda function because the data volume’s pretty small for that particular table.
Kevin Hartman: And you don’t have to turn on your data. Turn on compute to get to your data, in the way that some other systems actually have you move your data into a proprietary system. And once you do that, okay, now I need to query it. Okay, let’s turn on then. [laughs]
James Winegar: Oh, really? Okay, sure. And that’s my only option at that point.
Kevin Hartman: Yeah. And then if you want to run analytics on that or you want to share that with someone, let’s say you want to share that data outside of your environment, you got to turn on your compute. Whereas the way we approach that is you turn on the compute at the place that you need it, and the data’s there.
James Winegar: Well somebody has to turn on their compute. If you’re doing a share from some other systems, somebody’s responsible for compute. But then that compute is tied into that vendor ecosystem. [Right] You can’t use a different query execution engine in the default setting there, which is vendor lock-in. And how does vendor lock-in create things? I think this segues into the next base question that we had wrote, which was about interop between open source tooling and proprietary tooling. When we have these open source formats, which allows a lot of flexibility, and then Databricks’s data intelligence platform is basically a bunch of glue across these open source tooling that for the most part Databricks has built up. But you have these proprietary systems that you have to go talk to for various reasons, SAP’s probably a good example here, a lot of large organizations have a lot of their business data in SAP. SAP is a gigantic ERP that holds purchasing agreements, invoices, all these things that a lot of people care about, if you’re doing a forecasting flow. But Databricks has to allow the interop between those different systems. What are the challenges that you have in that situation?
Kevin Hartman: To help address those challenges between Databricks and those proprietary systems, a way forward is to rely on our ecosystem of connectors. There’s a huge number of ISVs and partners who have built those connectors for you to integrate with SAP’s environments or others, or Salesforce data. That is also something we support in something called Lakeflow Connect. It’s another set of tooling that is more native, it works out of the box. There’s a whole host of solutions that are available to you. You want to lean on some of those ready-made connectors so that you’re not creating custom solutions as best as you can. That’s not going to be the case for everything, but for some of the big ones that are out there, the more obvious connection needs, rely on that ecosystem to fulfill that piece. For organizations that may have already gone down a pathway of creating a data lakehouse themselves and then maybe it’s in a different format, maybe it’s using Hudi or maybe it’s using Iceberg. To solve for that, what Databricks has done is we’ve introduced something called Delta UniForm. That sits on top of the different protocols to treat them all uniformly, so that you can speak through the transformation layer and you perform a query and it just runs through directly onto your Iceberg tables or directly onto your Hudi tables.
James Winegar: It’s like middleware.
Kevin Hartman: Exactly. Other things for the connectors using Lakehouse, if it’s a source system and you want to federate and you want to govern, you can still do that through Unity Catalog as well through using Lakehouse Federation. And you can also do things with catalog federation, leveraging maybe a catalog that exists in AWS or other places, you can use catalog federation as well to promote that catalog description and make it available in Unity Catalog. And then there’s going to be some needs even from taking information from Unity Catalog making that available to some other system. That is also supported through open APIs and Unity Catalog is an open source technology now too. We talked about these proprietary systems and how do you create an integration between the proprietary system and an open source system, and most of Databricks is open source. I think another piece of that puzzle that we should talk about is legacy system migration into Databricks or modern data platform. There’s also a lot of downtime minimization you have to do in these types of workflows. Could you walk us through what a Netezza takeout might look like for Databricks? That’s a great question. How do we approach that? Our approach, I should say, our strategy includes a phased, end-to-end migration process. It’s a structured set of phases to approach your migration which would first include the start with a discovery and assessment and then developing a strategy, going into a production pilot, and then going through and concluding with the rest of your execution phases. The pre-migration is helpful in understanding your current data landscape. What are where could I have potential issues? How do I plan and prepare for secure access control when I’m moving things over? This helps me in anticipating and preventing issues that could cause potential downtime later. Then what we would move to is a production pilot. We’re taking a specific use case or a handful of use cases and we’re testing our migration strategy. This gets us more information and helps us identify potential issues in an environment that’s controlled and then inspected before going forward with the full migration. It’s also helpful for us to say, okay, we need stakeholder investment and buy-in and participation and it allows someone to see, ‘Oh, this is what it’s going to look like.’ So we’re addressing some of the people process changes as well when we get to the final end state. It also potentially will generate excitement. Because ‘Oh, this is really, really cool. Wait, can we go faster? Can I get more people on this? Okay, accelerate this because this is going to be really, really important as I move forward.’ Running production pilot is key for a number of different interests. Once I’ve started this journey, if I’m running a production pilot, I would like to profile and leverage some automated code conversion tools to help me with my migration process and I would want to first inspect how that’s going to perform in that pilot mode. What these tools do is they help to convert some legacy code, like stored procedures, ETL scripts to Databricks SQL equivalents and ensure that your business rules are preserved, and using tools like this can help with up to 80% of your migration work and then reserving some of the rest that might be a little more tighter or harder to resolve, more of the complex queries are done as the rest. Other things to consider in this process is doing parallel testing, and using data reconciliation tools to discover do my results match up, do my expectations match up. And then having those things in place as part of your general data quality checks. Those just become interesting then later for downstream data preparation to ensure that, am I running into some interesting data quality challenges later on? So you’re thinking about those as part of development of your parallel testing. Once all that concludes, you’re going into post-migration and support and monitoring, and leveraging our platform you can do a lot of that monitoring and observability and looking for things before they become problems. That’s an end-to-end, complete view. You want to consider doing this in phases though, so that you take, along with starting with that pilot, to further like, okay, what’s the next level of use cases that I want to support and moving to get to the things that are going to be delivering the most value for you fast and first. And setting up your plan that way. The other thing I would recommend is this could be challenging and we would recommend, of course speaking to why I get involved with, why I support, Databricks, is to really leverage the power and capability of those consulting and SI partners that have done this before. Leverage that expertise of the SIs, they bring a lot of deep industry knowledge, they’ve done this multiple times, and they can bring a tech proficiency to the process, things that they’ve seen that you may not have seen before if you’re going this alone.
James Winegar: What’s your philosophy on community building? Because I see a lot of it through our interactions with each other.
Kevin Hartman: Community building for me is shared passions in this space. It goes deep for me. It goes before Berkeley, before as a practitioner in software construction and consulting. That’s been my career, 30 plus years in creating solutions for customers, for clients. And really putting myself in their shoes as well, of how I would want my system to be built, but also to take pride of what had been delivered and how that solution is really impacting the customer or the stakeholder. But you also want your system or the thing that you developed to have shelf life and to be supported and to be maintained. All those sorts of things are baked into that philosophy of making sure that not only is the customer happy but you’ve got a maintainable solution consistent with the open standards and open protocols and open source that really drives my desire to be in this community and do community building, because I believe that we’re all at our heart, we want to be good practitioners. This is a really packed thing. Good practitioners, but also responsible practitioners in a number of different fronts. Responsible meaning I want the solution to be maintainable. But also, what we do happen to know in our field and what we have done in data science is to be responsible citizens as well and to have an eye out for what maybe what we should and shouldn’t do. There’s some of that aspect as to, just because we can build it doesn’t mean we should. [laughs] This is going back to some data science philosophical questions too. But all those things are baked in. You don’t necessarily know the answers to some of those things without community. You rely on community to help support one another, to test your own knowledge, but also to test your own philosophy. That’s why being involved in this space, going back to this early thing around having data silos, I don’t want to be people silos either, we should be a community. That’s what goes back to all the things that I’ve been involved in and continue to actively promote. You mentioned the LLM Ops initiative that we’re starting as a grassroots thing, involving multiple stakeholders of different people from different walks of life from industry, from academia, from our own internal developers. It gives people a chance to work together and it’s cross-community, it’s cross-collaborative, cross-domain. That’s where innovation happens too. I might be going in different loose threads, but they all tie together. Innovation is also part of this too. You don’t get to innovation without involving multiple people. You don’t get to the same level of epiphany when you don’t get to talk to people about their needs and their motivations and then helping to drive and move us forward.
James Winegar: I think it’s kind of like sports in a way where you need to practice with people who are roughly at the same level or nearby because it forces you to do more, it forces you to reach the next step. And then when you got a bunch of people, everybody’s feeding off each other, ‘Oh, I didn’t think about that use case because I work in my little bubble over here.’
Kevin Hartman: It is contagious. It’s really contagious. And then it’s fun, because there’s a definite spark and light when you’re working with like-minded people. This conversation that we had today. It’s just fun. Because this is the space that we live and we get to talk about. Really? With my job, the things I get to do… This is recorded, but maybe we should cut this part out. [laughs] ‘Wait, really? You want to pay me to do what I get to do?’ That’s awesome.
James Winegar: That’s the type of people that companies want to hire anyways. Because you know that if somebody really is about something, then you know they’re going to do good work. It’s inherent to, if they’re passionate, people will do good work. It’s just inherent to the process. If somebody doesn’t like what they do, it’s clock in, clock out, I’m doing exactly what I’m told, nothing more, maybe even less, because you don’t enjoy it. It’s a suffering. And a lot of people get stuck in a world where they have to do a job they hate, and it’s painful.
Kevin Hartman: It can be, it can be. But there’s a path for you. Seek out people who could inspire you to get beyond maybe that local proximity or local thing that you’re stuck in.
James Winegar: Working with other people is really important. Also, people quit their manager, not the company, typically. When you look at what does that mean? People care more about the people that are around than the whole organization. Because it’s the people who have bad interactions with their manager, that’s what really stops them from being happy.
Kevin Hartman: It’s true, it’s true. And luckily I work for an organization where what we talked about with the community, that’s the culture. And that goes up and down to the executive level, Ali Ghodsi, they’re also professors, adjuncts, and they’re also practitioners in this space, they know it. So when they’re having conversations with them, they can go up and down and navigate. That’s so cool.
James Winegar: It’s really interesting about Databricks in particular from the executive team because the vast majority of the executive team came from academia, they chose their roles based off of what they fell into during their early days. But they were all professors or grad students and things like that before. And so they all contribute to the community. It started as an open source project really, interesting idea about Spark, where it’s, ‘What if I relaxed the memory sharing just a little bit?’ That one idea now has evolved, it’s been I think 2004 or something like that, 20 years of just evolution of this idea started with just a simple idea.
Kevin Hartman: It’s a nugget before. Yep. And then the team surrounded it, they developed it, and I think 2013 is when the company became a company, I believe that’s when Spark was also open sourced, put out. So around that time frame, don’t quote me. The idea preceded that but the company itself was fairly new. It was a research project for a good long time before it got to a state where the community was active, Spark Apache, it was donated into Apache, probably 2011, 2012 time frame. The idea comes in and then they build up that idea and then fast forward, got Databricks, 15 years. Yes. And it’s one of the most heavily contributed open source initiatives around. And it’s a continuation of that philosophy that Databricks does, taking something that may have been developed internally and open sourcing it. Delta format, open source. Unity Catalog, open source. MLflow, open source. There’ll be more. There’ll be more innovations coming and those’ll be open source too.
James Winegar: There’s also just that evolution of Spark itself from just RDDs, resilient distributed data sets, to DataFrames, structured streaming, the catalyst engine, there’s a whole lot of innovation just on top of the engine itself.
Kevin Hartman: Right, yeah. Photon. There’s a lot of innovation that has been occurring behind the scenes making that seamless though for customers of this. You don’t have to think about it. Okay. So Kevin, where can people learn more about Kevin and Databricks? Well with Databricks, there is a website that a lot of information about Databricks can be found there. I would also be happy to talk again at any time if you’re more curious about what Databricks can offer to you or others that are maybe listening. About me, I should share that I’m coming up to be included in the registry ‘Who’s Who in America’ and I’ll be featured in ‘Who’s Who of Industry Leaders’. That’s coming out soon. If you really want to know more about me, about my bio background, more about my philosophy, that’ll be there, forthcoming. But also just reaching out or finding out more about me on LinkedIn, happy to connect and talk with anyone about what I get to do in Databricks and philosophy on data science, Gen AI, other things, happy to connect and talk more about that.
James Winegar: Awesome, Kevin. It was great to talk to you. Hope you have a great day.
Kevin Hartman: You too. Thank you, James.
Jason: That’s it for this episode of Data BS. If you enjoyed this episode, make sure you subscribe wherever you listen to your podcast and not miss the next one. This episode was sponsored by CorrDyn, a data consultancy that helps organizations unlock the power of their data. If you have a data challenge, we can help. Visit corrdyn.com, C-O-R-R-D-Y-N dot com, to learn more. See you next time.
