Listen on
Overview
Scientific research, particularly in biotech and life sciences, often struggles with inconsistent, non-scalable data analysis workflows. Relying on custom scripts hinders reproducibility, complicates collaboration, and creates significant technical debt that slows innovation and inflates R&D costs. Data leaders face constant pressure to deliver reliable results faster, requiring systems that can handle massive datasets while maintaining strict fidelity and controlling cloud spend.
This episode features Harshil Patel, Director of Scientific Development at Seqera Labs, the company behind Nextflow. Harshil brings over a decade of firsthand experience from leading research institutions like Cancer Research UK and the Francis Crick Institute, where he grappled with these very challenges. Now, at Seqera, he guides the evolution of Nextflow—an open-source workflow management tool—and the Seqera Platform, translating complex scientific needs into reliable, scalable data solutions.
Ross Katz and Harshil discuss how Nextflow and its community-driven nf-core initiative address the critical need for reproducible, portable, and collaborative scientific data pipelines. They explore how the Seqera Platform provides the operational layer for monitoring, cost optimization, and advanced data staging with tools like Wave and Fusion, allowing organizations to manage genomic datasets efficiently across diverse cloud environments while keeping costs contained.
Key Takeaways
Reliable Reproducibility Requires Software Engineering Standards, Not Just Lab Protocols.
Scientific data analysis needs more than experimental reproducibility; it demands consistent computational results. Nextflow integrates containerization (e.g., Docker, Biocontainers) and Git-based version control, ensuring pipelines deliver identical outcomes regardless of the execution environment—today or a decade from now. This approach shifts the burden from fragile, custom scripts to standardized, auditable workflows, which is critical for scientific validation and regulatory compliance.
Consolidating Analytical Efforts Through Open-Source Best Practices.
The nf-core community’s model, which enforces ‘one pipeline per data type’ and rigorous peer review, drives collective improvement in essential scientific workflows like RNA-seq. This prevents individual organizations from reinventing standard analysis methods, fostering a collaborative environment where industry players, including large pharmaceutical companies, contribute to shared, validated resources. This collective effort accelerates progress by building on a common, high-quality foundation.
Unchecked Cloud Parallelization Erodes Data Processing ROI.
While cloud infrastructure offers immense scalability for processing thousands of samples, excessive parallelization can rapidly inflate costs, particularly due to data egress and ingress charges. The Seqera Platform includes features like Fusion, which optimizes data staging by mounting buckets directly into containers. This reduces unnecessary data movement and API failures, enabling organizations to achieve high throughput without sacrificing budget control or pipeline stability.
Dynamic Workflow Control Outperforms Static Configuration for Complex Analysis.
Nextflow’s domain-specific language, built on Groovy, provides greater flexibility than static YAML or JSON configurations, allowing dynamic decisions and integration of existing programming libraries within pipeline steps. This enables bioinformaticians to address complex, evolving analysis requirements without needing to rewrite entire sections of a workflow. The result is simpler maintenance, adaptation, and more powerful analytical capabilities.
Related: CorrDyn excels in data engineering and building data reliability into complex systems for clients in biotech and life sciences. We also help organizations with data cost optimization.
Full Transcript
Jason: Hi everyone, this is Jason, producer of Data in Biotech. Before we get started, I wanted to let you know about our latest white paper. It’s a comprehensive guide to implementing machine learning models in biotech manufacturing. It’s a complete overview of all the potential problems of ML adoption and, more importantly, how to solve them. To download it, simply visit connect.corrdyn.com/biotech-ml. We’ve also dropped the link in the show notes of this episode. Okay, let’s get into it. Welcome to Data in Biotech, a podcast from CorrDyn where we explore how companies leverage data to drive innovation in life sciences. Every two weeks, we sit down with an expert from the world of biotechnology to understand how they’re using data science to solve technical challenges, streamline operations, and further innovation in their business. This week, we sat down with Harshil Patel, head of scientific development at Seqera Labs. Harshil shared his journey from working at Cancer Research UK and the Francis Crick Institute to joining Seqera, the company that brought Nextflow to market, a workflow management tool designed to simplify the complex process of analyzing large sets of biological data. During the interview with Ross, they both discussed the importance of collaborative open source projects in advancing scientific research and why it’s essential that as an industry, we create standardized, reproducible workflows. Here we go.
Ross Katz: Harshil Patel, welcome to the Data in Biotech podcast.
Harshil Patel: Thank you, Ross.
Ross Katz: To kick us off, would you mind giving us an introduction to your background and what brought you here today?
Harshil Patel: I’m Harshil Patel, director of scientific development at Seqera. I’ve got a bioinformatics background. I initially started working at Cancer Research UK and the Francis Crick Institute in London. I worked in a core facility there for over 10 years doing mostly genomics analysis, working with some awesome scientists, Nobel Prize winners at the Crick. I learned Nextflow very early on in its evolution, when it was still being developed out of the CRG in Barcelona. I became part of the furniture with the community and I’ve never left. I also started integrating myself with nf-core, which is this community that sits alongside Nextflow where they have best practice pipelines, and it’s become really popular for running pipelines, whether that’s off the shelf or contributing to the community. The natural transition for me after the Crick was to carry on working with Nextflow. I love Nextflow. It changed my life. And then to join Seqera, which is essentially the home of Nextflow. I’ve known Paolo and Evan for quite a long time now. Pretty much as soon as they got Series A funding, they said to me, ‘do you want to come on board as our only bioinformatician?’ And I couldn’t refuse. That was my next career step, which was almost now two and a half years ago. I joined as employee number 13 in the company. We’re now almost over 80 in that space of time, and I’ve started building my own team out. It’s been phenomenal being part of the growth of the company as well as Nextflow and all of the other cool things that we’re doing in the open source. My role at Seqera now is very much customer focused. We tend to be the content experts for Nextflow, nf-core, as well as the platform. We help our customers with their adoption of the platform and various other components, which I’m sure we’re going to drill into later on. I’m also still heavily contributing back to open source and our team is very much committed to that because that’s essentially where Nextflow came from.
Ross Katz: Could you give me an introduction to what Nextflow is and what Seqera is as a part of that?
Harshil Patel: Nextflow was initially written and developed to solve problems with scalability at the CRG. The CRG is one of the most awesome institutes that you’ll ever see for research. It’s located on a beach in Barcelona — the envy of the world. Nextflow was written as a workflow management tool. The idea is that when you do any type of analysis, it’s a series of steps. You take some input data and process it. That produces an output file, and bioinformatics is very file heavy — a lot of data flows through these pipelines. The output of one task goes to another as input, and you end up with this graph where data flows through the pipeline until you end up with something you can interpret as a result, whether that’s gene expression results or a single cell analysis plot or something you can view in a browser. Nextflow was developed to tackle these problems and to bring modern software engineering practices into the mix, which wasn’t really well known in biology at the time. Typically before then, I used to write custom scripts to do my analysis which no one else could ever run. Nextflow was written to solve some of these problems. It started off 10 years ago through the community. It’s open source, it will always remain open source. Over time it gained more and more traction in the life sciences, where others wanted to start using it for their own purposes. Five years into that, Seqera was formed initially to offer training. Evan and Paolo formed Seqera to offer training and go on pilgrimages around the world to train others in Nextflow. Then we had a very simple application at the time, previously known as Nextflow Tower, that allowed you to run and monitor Nextflow pipelines via a user interface — rather than running via the command line. Seqera and the platform we have now has evolved through that and we’ve got a ton of other features. The main aim of Seqera is to make scientific data analysis accessible at any scale.
Ross Katz: Can you talk about why people choose to use Nextflow in the first place? You mentioned writing a bunch of custom scripts versus leveraging the open source packages available in nf-core. I’m imagining it saves them a lot of time and effort in terms of writing custom code, but there’s also a reproducibility element. What are some of the considerations that lead people to the Nextflow ecosystem?
Harshil Patel: Nextflow has a number of key benefits that make it very appealing for scientists and anyone in general. That includes portability, for example. You can write a Nextflow pipeline that runs on your laptop and then quite easily run exactly the same pipeline on a complex infrastructure like cloud or HPC. Nextflow supports out of the box, I think up to 18 different executors already — common executors where you can run exactly the same pipeline in different situations. Scalability: Nextflow inherently takes care of things like task parallelism. When you typically run a pipeline on a scheduler and you’ve got 10 samples, running it on 1,000 samples is a completely different problem. You don’t necessarily want to get into the weeds of orchestrating the execution of all of those tasks. Each pipeline could be 20 or 30 different processes and each task is 1,000, so you start doing the maths and slowly you start seeing that it’s quite a complex problem trying to push data through a pipeline. Nextflow manages that inherently out of the box. One of the key things for me is reproducibility. Science is built on the concept of reproducibility. If we can’t do an experiment three times and reproduce what we’ve done, then there’s a problem. In a similar vein, when you want to analyze the data coming out of those experiments, we need to be able to reproduce that result over and over again. There are some things that Nextflow has adopted over time, like containerization, that weren’t very well known in life sciences but were used more in modern software development at the time. Scientists now know what a container is. You go back five or 10 years, no one knew what a container was or what it meant. It was just some abstract concept — ‘what’s a container? I put my takeaway in it.’ But it’s an isolated environment where you can reproduce exactly the same result from a pipeline execution. Those are some of the key benefits of using something like Nextflow. Part of this is also the shift to cloud computing. A lot of organizations are moving to the cloud to process their data, and I can bet your bottom dollar that there are a number of organizations that have started off with bash scripts, Python scripts, or legacy systems that no one’s maintaining anymore. Now they need to look for the next solution to run their pipelines on the cloud. Nextflow has matured to run workflows on the cloud — this was forward thinking on Paolo’s part, adding cloud support and really working with AWS, Azure, GCP and other cloud providers to improve support for cloud execution. That’s made it easier for those in Big Pharma and biotech to transition to the cloud and start using off-the-shelf content like nf-core because it works out of the box.
Ross Katz: That makes a lot of sense. What are some of the biological analyses or processes, pipelines and use cases that are available or typically used inside of nf-core that people can just off the shelf pull down and start working with?
Harshil Patel: nf-core now has over 100 pipelines. One of the big advantages of the community is that you can bring your domain expertise to nf-core. We typically only allow one data type per pipeline, so there’s only one RNA-seq pipeline, there’s only one ChIP-seq pipeline. If you allow multiple then slowly things get out of hand — it becomes confusing as to what you need to contribute to and how. There are some fundamental guidelines on nf-core, like the testaments of nf-core, that have been around for quite a while now. They’ve helped to define what the community looks like and how we can impose certain restrictions and rules in terms of pipelines. We’ve got over 100 pipelines. The most popular is probably the RNA-seq pipeline, which I still maintain and manage. It tends to be the gold standard pipeline for nf-core in terms of RNA-seq processing, but also in terms of what we’re doing with Nextflow. The community isn’t just about doing analysis for a particular data type — it’s also about how we can push Nextflow, how we can evolve it to make it even better and solve some of the problems we’re not necessarily solving properly by implementing a pipeline in a typical way. Whatever we’re doing on nf-core gets filtered back as feedback to Nextflow and Paolo, who I can just DM and say ‘Paolo we need this, can you help us out?’ And he finds a solution with others like Ben on the team who are doing a phenomenal job with the Nextflow codebase. RNA-seq tends to be the most popular, but there’s all sorts of pipelines now. Genomics is one of the most common applications — there’s just tons of genomics data. But on nf-core we have genomics pipelines, proteomics pipelines, metabolomics pipelines, image analysis pipelines, protein folding pipelines, astronomy pipelines, earth sciences pipelines. This shows the versatility of Nextflow. Any piece of software you can put in a container to run a pipeline, you can have a Nextflow pipeline with that. It doesn’t have to be restricted to genomics, and we’re seeing more and more uptake of these sorts of things in the community.
Ross Katz: I’m really interested in the decision to constrain the pipelines in nf-core to one per analysis type. In the context of scientific analysis there’s this explosion of people working on different pipelines and competing ideas about how pipelines should be structured, and forcing the open source community to work together and figure out the best version of that pipeline that incorporates all of those competing paradigms makes a lot of sense. Could you walk us through that decision and how Nextflow’s open source model has shaped its development over time?
Harshil Patel: Nextflow offers benefits in terms of collaboration as well, which we haven’t really touched on. The way that you write Nextflow and make it available to others is via a Git repository, whether that’s GitHub, Bitbucket, or GitLab. When you write your pipeline, you commit and push it to a Git repository. nf-core itself is all GitHub based — we’ve got 100 pipelines, all of them on GitHub, all open source. All of our collaboration happens via GitHub. This is something that wasn’t necessarily around 10 years ago before Nextflow, which has essentially enforced that scientists and developers actually collaborate on writing code, and everything is version controlled down to the individual commit. When you add a piece of documentation and commit it on GitHub, you know exactly who pushed that commit. It’s not just a save on a local computer where you have no idea how that was done. Reproducibility is also important when you’re writing your pipeline, not only with the containers. When you write a pipeline, you release a particular version that becomes a static version you can run now or in 10 years time and it should give you the same result if you’re using the same containers and the same input data. The open source model that Nextflow offered to collaborate via GitHub was essentially the decision to then also form the nf-core community around GitHub and allow people to communicate and collaborate on these pipelines. Writing a pipeline is one thing, but reviewing code is also really important. Another guideline in nf-core is that before you release a pipeline you need to have it reviewed, depending on what your pull request looks like and where it’s going, you need to have it reviewed by other community members to ensure that you haven’t missed anything. People miss things all the time, including me. It’s good having oversight. Ultimately in terms of one data type per pipeline — and we’ve had pushback on this — there have been others that have said ‘we want another step in this pipeline that you currently don’t have.’ The way we address those problems is, ‘look, this is a community decision, no one really owns this pipeline. It’s owned by nf-core, it’s a community pipeline. We will make this decision jointly and be sensible. We’ll try and follow literature and what’s best practice and add it as an option.’ For RNA-seq processing, you don’t want to add 100 tools just because 100 tools exist. They might not be the best tools — you want to narrow that down to what is actually useful. In terms of contributions, it makes it a lot clearer to community members to say ‘we’re contributing to one RNA-seq pipeline, not 15.’ That avoids conflict. People that want to run a pipeline see all of their efforts going to improve a single pipeline rather than a ton of other pipelines that don’t make sense. There’s always one pipeline that just gets better and better over time.
Ross Katz: Is there like a hierarchy of community managers? Like a person who oversees each pipeline and then like a group of people who are nominated to be reviewers of the pipelines and assess whether this is worthy of incorporation, recommend that the pull request get merged in to the pipeline?
Harshil Patel: Typically community members will come in and suggest they want to add a proteomics pipeline. I, for example, don’t have any proteomics background. But other people come into the community because they like Nextflow and they want to contribute a pipeline to the wider community to use. They’re also happy to follow the guidelines, which is quite important. They’ll come to us and say ‘look we have a pipeline.’ Sometimes they may already be tag-teaming with others internally within their team or the community to get this pipeline off the ground. They’ll start working on it, get a plan of action. We have a request review channel in nf-core where members of the community can request reviews on their pipelines from others. We have a tag-teaming approach there where members will swap reviews with others if need be to get their pipeline through. It’s fairly dynamic. Some pipelines have set members contributing to it and a lead developer. Others are more organic where you’ve got 15 or 16 active developers contributing to that pipeline, like nf-core sarek, which is a variant calling pipeline. It’s a huge pipeline and over time loads of people have started contributing to it routinely and have formed their own team to improve the pipeline.
Ross Katz: That makes a lot of sense. So what are some of the biggest challenges that people face when onboarding to Nextflow for the first time? Like individually or as an organization?
Harshil Patel: Nextflow at times can be like Marmite. You either love it or you hate it. This is sometimes the draw to Snakemake for people — it’s written in Python, it’s got the familiarity that a lot of people have with a conventional programming language. Having said that, there are other benefits that Nextflow offers that make it a really good entry point. You don’t necessarily need to know any Java or Groovy. There’s a common misconception that because Nextflow is written in Groovy, you need to know Groovy. Having worked with Groovy, I can say it’s amazing to write Groovy code. I came from a Python background, but having learned and written Groovy to do some of the more mundane things that aren’t necessarily exposed by the Nextflow DSL has been valuable. Just for background, Snakemake’s written on Python, it’s a bit like a Make type of solution, whereas Nextflow is written on Groovy. Groovy is a very powerful language to write what are called DSLs or domain specific languages. That’s partly why Paolo at the time chose to use Groovy as the backend — he also liked Java, and Groovy is essentially built on top of Java. It’s like a Russian doll situation: you’ve got Java, you’ve got Groovy, and then you’ve got Nextflow on top. Nextflow has its own syntax and its own methodologies to write pipelines and stream data through. If you do want to get into the weeds and do more complex things, you have the option of doing that. That’s one of the biggest benefits of Nextflow compared to other languages that might be using static formats like YAML — with Nextflow rather than having a standard flow, you can do more dynamic things in between tasks by leveraging standard programming languages’ power underneath.
Ross Katz: Having it built on Groovy, having that custom DSL that was developed as part of Nextflow sort of gives you the flexibility in between those process steps to make a lot of decisions about how the pipeline runs.
Harshil Patel: You can also import existing libraries. Any programming language has tons of existing functionality that you can just import as a library. Whether that’s to load a YAML file or a JSON file, there are existing libraries in Java or in Groovy for that, and you can essentially piggybank all of that existing functionality within the context of a workflow management tool without having to rewrite it all from scratch.
Ross Katz: Yeah, and I would imagine with the advent of generative AI and coding assistants that writing Nextflow pipelines is easier than it’s ever been.
Harshil Patel: There are still some people sitting on the wall about how good it is — but that’s generative AI in general. It’s something that you need to review and make sure that it’s doing the right thing. It’s getting closer and closer in terms of generating these sorts of processes. It depends on the context in terms of what you ask it to do. There may be a need to curate the result and inspect it to make sure it’s getting it right. But with everything else, it’s changing the world.
Ross Katz: That makes a lot of sense. Beyond Nextflow, can you just introduce us to the other components of the Seqera ecosystem?
Harshil Patel: Nextflow is essentially why Seqera was founded. It’s a language that allows you to write pipelines, it’s open source, you run it typically via the command line. What we’re doing at Seqera is trying to build a platform where Nextflow is essentially the engine. The platform itself is a user interface. It’s now called Seqera Platform — it was previously called Nextflow Tower. We’ve been through a massive rebrand effort. People are still slipping up, including me, but we’ll get there one day. We’ve built a platform that uses Nextflow as the engine under the hood. The platform allows users to make themselves more efficient in terms of running their Nextflow pipelines, running their analysis, managing their users, and hooking in complex cloud infrastructure — which via the command line with Nextflow has been quite painful. The platform has features built in that allow you to hook in cloud compute very easily. One of the biggest advantages of the platform is that you run the Nextflow pipeline on your compute using your storage. The platform is a pane of glass for you to run Nextflow in your environment. You keep your own data, you manage your own compute, and you manage your own discounts with your cloud providers. We don’t ingest your data — it stays with you — and some of our customers really love the fact that they can use their own compute when running Nextflow. There are also cost-saving features in the platform to optimize pipeline executions and bring down costs in terms of how much things end up costing in the cloud, which can get out of control very quickly. It’s a different problem to HPC. With HPC you never see the light bill, but with cloud providers you need to somehow control it. The platform helps with these sorts of things. We’ve built a number of products on top of Nextflow — most of which are also open source. MultiQC, for example, is a bioinformatics reporting tool used commonly at the end of pipelines to aggregate QC results from various tools. That’s also now a Seqera product and it’s open source. We’ve got a containerization solution called Wave that makes it a lot easier to provision and build containers without having to manually do this yourself. It’s got a number of other benefits that make it seamless to provision containers for your Nextflow pipelines. That’s also recently been open sourced. We have another solution called Fusion, which is a very powerful solution for optimizing the way that you’re interacting with storage, especially in the cloud context. In the cloud you’re typically dealing with blob storage — remote storage where your files sit in a bucket. You need to make them available to Nextflow to run the computation, whether that’s a fastQ file or a reference file. Typically what happens is you copy that file into the virtual machine where that task is running before you can perform the execution. Then once you’ve generated an output file you copy that out to another bucket. That can be quite wasteful because you’re moving a lot of data around the cloud. What you really want is to bring the data into the virtual machine seamlessly, and that’s exactly what Fusion does. It mounts the bucket within the container for you to perform the task execution without having to copy and stage it in, avoiding API issues and failures from moving a lot of data around the cloud. Fusion has become a really nice solution for making these pipelines a lot more stable in the cloud, especially when you’re looking at larger datasets. That’s the product portfolio we’ve got now. The platform also isn’t just about running Nextflow pipelines. We’re taking it to another level where the Nextflow pipeline portion is maybe the secondary analysis where you take fastQ files and generate some counts. We’ve now got a feature called Data Studios that allows you to take that output and interact with it in a Jupyter notebook environment and various other things. We really want to see the platform as the modern biotech stack. You can take your data, interact with it, and manage your organizational needs all within one user interface. There’s also an API as well as a command line interface specifically for the platform, so if you’re a bioinformatician that loves automation you can hook into that as well.
Ross Katz: That makes a lot of sense. You’re taking Nextflow, which is exclusively for running your pipelines in an intelligent, parallelizable and reproducible way, and you’re trying to build an entire data platform around it where you’ve got the user interface component for cost monitoring, cost optimization, and I’m assuming performance optimization as well. One of the challenges with Nextflow is that yes it’s infinitely parallelizable, but that is both a benefit and a drawback — you can have a situation where your code isn’t running as performantly as it could so it’s having to be parallelized more than is optimal, or it’s parallelizing so much that it’s costing you much more than you would like from a business perspective.
Harshil Patel: It all depends on how you write your pipeline. There was a really good talk by Hatem who’s at Google — I’ve known him for ages. He’s been working with us on Nextflow support for Google Batch and various other things. He gave a really good talk at Nextflow Summit a couple of years ago where he showed that if you take a fastQ file and fragment it up too much in a cloud environment, costs start ballooning. You can quite easily do that on an HPC environment because it makes sense — you don’t see the light bill with HPC. Whereas you do with cloud. There’s always a sweet spot that you can optimize. Simple things like splitting a fastQ file: the more splits you have, the more virtual machines you need to provision, and the more it’s going to cost you. There are ways to optimize the way your pipeline is working to bring those costs down. The platform itself has a number of features built in that allow you to see how long a task took, costing, and various other things to get an idea of what may or may not be efficient.
Ross Katz: Yeah, I’m interested in Wave as well, so that’s the containerization solution. What are some of the features that it’s layering on top of something like Docker that add value for a workflow like this?
Harshil Patel: Wave has a number of features. Initially Wave was actually developed for Fusion. Nextflow runs in the cloud and will run every task in its own Docker container. That container contains all of the software it needs to run that particular task, whether it’s BWA or fastQC — they’re all in that individual container. When we developed Fusion, it’s a binary. We don’t want to rebuild all of the containers in the world because these containers have already been built. Biocontainers has an amazing resource of hundreds of thousands of containers for every single Bioconda package. We leverage that quite heavily in the nf-core community as well, so we don’t have to maintain and manage these containers ourselves. The solution we found is that we implemented Wave to use these existing containers and inject a layer — in this case the Fusion binary — into the container at runtime, so that the binary becomes available to the container without having to rebuild it from scratch. This became a solution for Fusion — it was actually a byproduct of Fusion. But since then it’s gone on to have its own functionality for various things like authenticating with private repositories. There’s also a ton of other stuff we’re working on that are going to be game changers, because Wave will also make it very easy to provision multi-tool containers. Right now there are a lot of single tool containers, but we’re building out functionality — which we’re going to announce at the Boston Summit — that allows you to provision these containers on the fly for multiple tools with just a click of a button. Picking what you want, build me a container, no fuss. Wave can offer a lot of benefits in terms of making it easy to build and provision containers so you don’t have to manage all of that yourself.
Ross Katz: Prior to running the pipeline, you have to build all of the containers. All of the containers contain all of the dependencies for the analysis. Those containers are independently managed by different people who are responsible for making sure that the analysis embodied by that container runs the way that it’s supposed to, that there are no dependency conflicts, that there’s no bugs in the way that it builds. And so it sounds like Wave is like a layer on top of that that allows you to incorporate into this diversity of containers a set of tools that are necessary for the entire Nextflow ecosystem to run as it’s supposed to end-to-end. And in particular with Fusion, you’re managing the way that files get moved around since that can lead to both latency in terms of the amount of time it takes for data to move from one place to another, but also cost, especially when you’re operating in the cloud, there are fees for moving files in and out of places. So having a solution like that allows you to take any container and optimize the way that data is managed within that container has both time and cost benefits.
Harshil Patel: Absolutely. On the container provisioning side with Wave, there are benefits on the developer side because when you’re writing a pipeline and you need a container, Wave makes it very easy to provision that container. Behind the scenes we’re just using the Conda APIs to query which packages you want and then build that container on the fly and host it for you so you can just pull it with no fuss and include it in your pipeline. That’s a massive cost saving exercise. As you said with Fusion — ingress and egress can end up with huge charges on cloud providers if you don’t carefully assess what you’re doing and try to stick to a particular region where you want to process your data. Fusion will help with that because it means you have to move data around a lot less, but you’ll also see a lot more stability in the way your pipeline’s running because it’s not erroring out whilst it’s trying to download a file. There are various other IO speedups and things we’ve optimized with Fusion as well.
Ross Katz: What are some of the most complex or amazing things you’ve seen done inside of the Seqera ecosystem?
Harshil Patel: Some of these big features like Wave and Fusion coming to fruition has been a process. It was something that Paolo dreamt up — that we need solutions that make it a lot more stable to run Nextflow pipelines especially in the cloud. Most of our customers are on the cloud, 70% of them on AWS, with growing adoption on Azure, GCP and other cloud providers. This is one of the great things about the platform: it’s cloud agnostic. In the similar vein as Nextflow, it can run on any platform. You can hook in your compute no matter where you are and in which region you’re running, so you analyze the data where it lives and you don’t need to shuffle it around. Watching Fusion and Wave come to fruition has been really cool in terms of the engineering effort and how much time and benchmarking and optimization we’ve put into it to see the benefits. At Seqera one of the key things is that we’re encouraged to think outside the box and that mentality is nurtured in terms of pushing the boundaries, using the next technology. That’s somewhat evident in the way that Nextflow was developed and how it’s changed the game in life sciences. We’re continuing to try and push the boundaries in terms of what the next best things are likely to be. Fusion and Wave were very hard problems to solve on the infrastructure side, especially in the cloud, and they’re now reaping benefits.
Ross Katz: And what about on the user side, on the on the companies that are leveraging Nextflow? Have you seen any sort of amazing pipelines or biological analysis that have come out that have blown your mind in terms of what Nextflow has enabled them to do?
Harshil Patel: We have customers that are running Nextflow on thousands of samples at a time. One of the great things about the cloud is that yes, if you don’t keep a note of costs it can end up costing a lot, but if you do want to run 10,000 samples at the same time through an RNA-seq pipeline you have the ability to do that. There have been times where I’ve been on a customer call and they’ve had a blow-away comment like ‘oh we ran 10,000 samples through this pipeline’ and I’m thinking ‘you did what?’ It’s not cheap either, because you want to perform a series of steps on the same sample to push it through a pipeline, which costs money. But they have commitments and it makes sense. There are providers like 23andMe who are customers running millions of tasks through Nextflow, and seeing customers push the boundaries of what Nextflow can do is refreshing — and it’s actually a challenge for us to solve and make the product even better.
Ross Katz: Awesome. It seems like this paradigm of scientists engaging in like collaborative tool development and data sharing is a way of going about science that I think just makes a lot of sense and leads to more progress through that social engagement component. So I’m curious from your perspective, since you’re at the top of this very large community doing this kind of work, how do you see the future of biotech, genomics research evolving?
Harshil Patel: Community has from the very beginning been important to Nextflow. It’s also now incredibly important to Seqera. A big chunk of my team’s time goes into improving nf-core pipelines and Nextflow pipelines, and a big chunk of my time is thinking about how we can improve what is happening in the community to keep it moving forward and evolving. That’s not going to change. One thing that we have seen change quite markedly since Nextflow’s come around, and even nf-core to some extent, is that biotechs and especially large pharmas are now more open to contributing to open source. Before Nextflow was around, everyone — including ourselves — had in-house Python scripts or bash scripts that were performing a task but you couldn’t share them with your collaborator across the pond because they wouldn’t work, they’d break instantly. All of the software dependencies and the tools and the hard-coded paths would just break instantly. Whereas now with the way that Nextflow is developed openly on GitHub, if these resources are publicly available, everyone can see what you’re doing and how you’re doing it. This has really encouraged and nurtured bioinformaticians and scientists around the world to adopt these practices internally. Also coinciding with biotech and large pharma adopting cloud, all of a sudden you’ve got a cloud platform that you need to run pipelines on but you don’t have any pipelines to run. So you look for a workflow manager like Nextflow that can run anywhere, and then you look for content on nf-core which should in theory run anywhere. Rather than reinventing the wheel and writing their own pipelines, a lot of these organizations are now adopting nf-core pipelines and saying ‘we just want to use this off the shelf, there’s already a good community pipeline that’s been validated and we can run it on AWS, why do we need to write our own?’ There’s a big shift in mentality. Doing things in open source has really changed the mindset in terms of how organizations that typically would do a lot of these things behind a firewall are now actively wanting to contribute back to open source. We have a number of professional services contracts with some of these big organizations where we have agreement with them to go back and contribute to open source. They’re using these pipelines but everyone is benefiting through our contributions back to open source. This is something that really is different compared to what would have been done about 10 years ago.
Ross Katz: Yeah, it strikes me that this sort of coincides with a change in viewpoint on the part of these big pharma and biotech companies about what their competitive advantage is, that it used to be that they viewed the the code for doing the analysis or the methods that they were applying to the data as being their competitive advantage, but now they’re viewing the data itself as their competitive advantage. And so protecting the data is paramount but the methods that they’re applying to the data can be something that’s more of a community-driven effort that that makes a lot of sense.
Harshil Patel: They still might have their proprietary pipelines that they keep private, but that makes sense commercially because it’s their IP. On the whole, for some of the more mundane analysis that you need to do like RNA-seq analysis, it’s run-of-the-mill now — quite well established. Single cell arguably is getting there too. For all of these off-the-shelf pipelines, they see the benefit of using open source pipelines and even contributing back to them so everyone benefits overall.
Ross Katz: Real quick before I let you go, is there anything coming down the pike for Nextflow or Seqera that you’re excited to talk about here?
Harshil Patel: There’s always lots. There’s not a dull day in the job. We’ve got a summit coming up in May and we’ll be announcing a ton of new features there. We’ve got another summit coming up in October in Barcelona, which is our flagship event, with lots of different features coming out as well. We’re trying to keep up with the game, constantly iterating and listening to users and customers to improve the product and build good relationships. On the roadmap for Nextflow, there might be a third version of the language, which is slightly scary but also really exciting because it’s going to solve a lot of the issues that have arisen over the years with Nextflow in terms of feedback from customers. We’ve taken that feedback really seriously and are going to iterate on the language to improve it for a number of reasons — troubleshooting, cleaner syntax, static typing, and various other things that would be nice to have. Through Seqera we’ve got the ability to improve the open source layer of Nextflow as well as all of our other open source products. Improving Nextflow will inherently improve the platform too. We’ve got other features like Data Studios for interactive analysis, Data Explorer for exploring data, and a host of other features that will be coming out, as well as improving the current features on the platform to make it even easier to run Nextflow wherever you want and collaborate on your analysis together.
Ross Katz: Well, Harshil, it’s been a pleasure to have you on the podcast. Really appreciate all the insight you’ve shared.
Harshil Patel: Take care. Thank you, Ross.
Jason: And that’s it for this episode of Data in Biotech. If you enjoyed the episode, please subscribe, rate, or leave a review in your podcast platform of choice. See you next time.
