Listen on
Overview
More than half of preclinical research fails to reproduce, and the root cause is rarely exotic: it is incomplete, inconsistent documentation of how experiments were really run. Scientists pull a protocol from a paper, try it in their own lab, and spend weeks troubleshooting because the critical details were never written down. Data scientists inherit the same problem a step later, when an anomaly in a dataset forces them to reverse-engineer what changed between last week’s run and this one. The tax shows up as lengthened timelines, wasted reagents, and ML models trained on data no one can fully trust.
In this episode, host Ross Katz talks with Harry Rickerby, co-founder of Briefly Bio, a platform that helps scientists capture, share, and collaborate on experimental methods. Harry describes it as GitHub for wet lab science. Harry was the first employee at LabGenius, where he built protein engineering workflows from scratch and watched basic assays fail for reasons no one could pin down. His co-founder Katya brought the computational view: datasets where something looked off, and no record to explain why. The conversation covers how Briefly uses LLMs to turn free-text protocols into structured methods, why biology’s reliance on proxy assays makes metadata non-negotiable for any ML ambition, where LLMs hit their limits inside specialized scientific work, and how version control and experimental lineage could finally answer the “why do we always add salt to this media” questions that hide inside every lab.
Key Takeaways
Inconsistent experimental documentation is a reproducibility and data-analysis tax across biotech
Scientists skip detail because the fully documented version (think Nature Protocols) is too dense to be useful in the lab, and the lightweight version omits the parameters data teams need to interpret results. Briefly’s bet is that the right tool changes that tradeoff by structuring information on the back end while keeping the scientist’s experience light. Until documentation is both complete and consistent, hidden variables like reagent lot numbers and ambient conditions cannot even be investigated.
In biology, metadata is the signal, not overhead
Unlike image recognition, where the source of an image does not change what it depicts, biology’s experimental data is a proxy generated by a specific procedure. Change the procedure and the data tells you something different about the biology. Combining datasets for large-scale analysis or ML only works if the metadata captures exactly how each dataset was generated. Otherwise the model is learning noise from procedural drift, not biology.
Use LLMs for narrow, integrated jobs—not as a chatbot bolted onto the product
Briefly leans on LLMs for what they do well: converting unstructured protocols into a structured schema, drafting boilerplate steps for common methods like PCR, and interpreting biology’s acronym-heavy language. Harry is explicit that asking an LLM to design a novel experiment is unrealistic today—the scientist’s context is too large to fit in a prompt. The useful pattern is narrow features that remove grunt work so scientists stay in control of the creative parts.
Full Transcript
Jason: Welcome to Data in Biotech, a podcast from CorrDyn where we explore how companies leverage data to drive innovation in life sciences. Every two weeks, we sit down with an expert from the world of biotechnology to understand how they’re using data science to solve technical challenges, streamline operations, and further innovation in their business. This week, we’re excited to be joined by Harry Rickerby, co-founder of Briefly Bio, a company that helps scientists capture, share, and collaborate on their experimental methods, kind of like the GitHub for experimental science. During this conversation, Harry shares the origins of the Briefly Bio platform, stemming from his frustration to get even the most basic assays working reliably in the lab, and how this resulted in developing a solution to improve experiment documentation and reproducibility in science. He also shares how large language models are used to facilitate experiment protocol structuring in the platform. His perspective on the limitation of LLMs in highly specialized contexts like scientific fields, and how metadata is generated within the experimental design process while interacting with an LLM. Here we go.
Ross Katz: Harry Rickerby, welcome to the Data in Biotech podcast.
Harry Rickerby: Hey, great to be here. Thanks, Ross.
Ross Katz: Awesome. Just to kick us off, would you mind just giving a brief background on what led you here?
Harry Rickerby: Sure. I come from a biology background originally, studied at Imperial. I got really excited about synthetic biology, about this idea that you could engineer biology to create meaningfully useful things for humans. I met someone called James Field during my undergrad there and did a project together and then I joined his company called LabGenius back in 2014 as first employee. We were focused on using data and machine learning to help with protein engineering, drug discovery. My role evolved as I was there. I started out in the lab and then moved into more leadership roles. The focus there ultimately for me was looking after what workflows we were going to bring into the company in the lab side and then how that interfaced with technology. What software we would build, what automation we would use, how we’d use data science. I founded Briefly with a couple of other folks last year and I guess we’ll talk a bit more about what that is as we go through. That’s how I got here.
Ross Katz: Yeah, awesome. Since you mentioned it, why don’t we just jump right in? Can you just provide an overview of the platform that you’re building?
Harry Rickerby: Sure. I guess the problem that we saw inside the companies that we worked at and then also more broadly in the field in general is this incomplete, inconsistent documentation of experiments. That causes problems all over the place. If you’re a scientist wanting to run an experiment, you don’t know exactly how somebody else ran it so you have problems with reproducibility. If you’re a data scientist wanting to analyze a dataset, you need to know exactly how that data was generated in order to be able to analyze it properly. All of these kinds of things. The idea behind the company was how do we solve this problem? How can we get to a place where when you look at an experiment, when you read the documentation behind an experiment, you know exactly how it was run. That’s ultimately the problem we’re trying to solve. The way we’re doing that is building a platform where scientists go to create and design their experiments. We take a lot of the grunt work out of that process and do a lot of the structuring behind how to make that information consistent, how to fill in all of the gaps that exist in a lot of the documentation that’s out there.
Ross Katz: It sounds like a really interesting platform. Did the genesis of Briefly come from some of the problems that you experienced while you were at LabGenius? What kind of things did you notice that allowed this idea to come into your head as the next thing you wanted to work on?
Harry Rickerby: One of the key things that I faced because I was early in the company, we were starting from scratch trying to build this really ambitious platform for protein engineering. We had to set up everything from scratch. We built our own lab and we were getting all of these workflows off the ground from scratch. A lot of the time what frustrated me was how difficult it was to get even basic assays, basic workflows working reliably in the lab. Even things that you would think, oh, I’m just pulling this in, it’s pretty standard, there’s plenty of papers published on it, I know people who’ve done it before. You get it into your own lab and then you try and run it and then it doesn’t work and you get into this endless cycle of troubleshooting and trying to figure out how things worked. Your timelines just lengthen and lengthen, especially for someone like me, I was super inexperienced, super naive. People who’ve done this before would know to build in a lot more wiggle room for those kinds of issues. That was my frustration and basically just not knowing really how these experiments were, how these processes were really being run. You read a paper and there’s just not enough information there to get the answer. Often you’d have to actually find someone who’s run it before, even then you still have problems. From the data science side, Katya, my co-founder, was coming at it from this position of okay, she was purely computational at my last company and she was getting a lot of data coming in from the lab. You run your analysis and then you see something in the data that looks weird. Figuring out why things look weird, they looked normal last week, why do things look different now? You have to go down this rabbit hole of figuring out exactly what changed in the experimental process. Was it something in the experimental process? Was it something else? Figuring out the answers to those questions is a lot of what spurred on founding this company, those frustrations.
Ross Katz: That makes a lot of sense. Can you give an overview of how the platform works from the perspective of a scientist who is running a given experiment?
Harry Rickerby: The scientist really, we’re focusing on that experimental design phase. Thinking about what exactly is it that I have to do as a scientist? I’m thinking about my experiment, thinking about my conditions, I’m thinking about my methods. What exactly do I have to do when I get into the lab? What are the steps I’m going to have to follow? What are the things I’m going to have to be tracking? We’re focused on how do we take that phase of a scientist’s workflow and make it as streamlined as possible? Often what happens now is a scientist will think about their experimental design, jot a few things down, think about the key conditions, and then run the experiment. Later on, they might go back and write up the experiment and think about okay what are all the things I did? Can we get to a place where scientists will design their experiments in this tool, get all of that information into this tool as easily as possible about their experimental design, help them think through all of those steps, give them additional information about maybe there are potential pitfalls in their experimental design, potential issues. And then also walk them through okay, now we’re going into the lab, what are the steps I have to take, where, when am I going to be tracking certain bits of extra information, bits of QC, all of that kind of thing. It’s about thinking about my experiment, thinking about the steps, getting all of that stuff into one place, and then running the experiment and following all the steps there.
Ross Katz: That makes a lot of sense. So why do scientists not document all of those things today?
Harry Rickerby: There’s a lot of variability, I would say. Some scientists do document a lot of stuff, sometimes scientists document less. The honest truth is that documenting every single step or every single parameter associated with an experiment is a black hole of complexity. It’s really hard to know what is the right line to draw, how much detail is relevant. An interesting part of it is the more you document often the less useful that artifact you create becomes. I don’t know if you’ve ever come across a Nature Protocols paper, these are the most incredibly complex, detailed protocols you can imagine. Every single step is meticulously documented. But getting an overview of a protocol like that is incredibly difficult. There’s this interesting trade-off that a lot of scientists are making where they don’t really want a full SOP, they just want here are the key factors that I need to remember. I’ve done this experiment before. It’s a combination of it’s a huge amount of work to document things and then even if you do document things fully you’re ending up with something which is less useful at the end of it, especially when you’re going into the lab. For us, it’s about how do we make that documentation process really fast and easy? How can we shortcut all of the steps to getting all of this information down? And also, how do we show it to a scientist in a way that gives you an overview, doesn’t drown you in the detail, and you can provide yourself with different ways of viewing that information.
Ross Katz: Interesting. I imagine that there are benefits to the R&D team as a whole in terms of the visibility into what each other are doing and how you collaborate on things. Are there features of the platform that are focused on the R&D team experience?
Harry Rickerby: Absolutely. That’s ultimately what motivated our development of this platform. How do we improve collaboration between scientists, between data scientists, between teams? That’s all part of it. The real utility of this really good documentation is when you don’t have everything in your head anymore, whether that’s you in six months or whether it’s a different person in the team. Part of the aim here is about consistency. It’s not just about documenting everything, it’s also about how do you make that information as consistent as possible? If a data scientist wants to say I want to query this experimental database and I want to see all of the experiments that were run with these particular parameters, they can search through it. It’s not just buried in a blob of text in an ELN entry or in a notebook somewhere. That’s all part of the plan. The first thing for us, though, is how do we make a tool which scientists really love to use and they like to document all of that stuff? Because until we have something which is really compelling for a scientist to use for themselves, you’re ultimately just going to be having managers telling their scientists do X, Y, Z. And if the tool isn’t useful, if it isn’t intuitive, they’re probably going to use other things and then you have the same problem again, information starts to spread into different places and then you don’t have everything in one place.
Jason: Are you a biotechnology company looking to unlock the potential of your business data? CorrDyn can help. We’re an enterprise data specialist that helps companies working in life sciences make smarter, strategic decisions. From developing the right data strategy that starts with our data maturity assessment to building and delivering bespoke technical solutions, we are equipped to tackle the most complex data challenges. We have partnered with dozens of high-growth organizations from manufacturers of custom oligonucleotides to molecular diagnostic companies to achieve data competency. Whether you need to supplement existing technology teams with specialist expertise or launch a data program that lays the groundwork for future internal hires, you can partner with CorrDyn to unlock the potential of your business data today. Simply visit connect.corrdyn.com/biotech to learn more. Now, back to the show.
Ross Katz: It seems like one of the goals of Briefly is to create this great experience for the scientist in constructing this experimental procedure, provides them the flexibility that they need to go in whatever direction they believe the experiment needs to go, but you’re also trying to capture all of this metadata, this standard information about how the experiment was run so that the data team can use that information later on. I have two questions. One is why is the data team so focused on getting that consistent metadata? The second question is how do you balance the needs of the scientist in terms of making it a great experience with getting the data team what they need?
Harry Rickerby: The question of metadata in biotech is a really big one and a really interesting one. There’s a big clamor and I’m hearing it more and more in the space for we need better quality data. In biology, we need better quality data, but also more data, higher throughput. In order for us to understand what’s going on in biology, but also if we want to use machine learning, use AI in order for us to unpick some of these really complicated biological relationships. The interesting thing about biology and biotech research in general is that the data is not ground truth. The vast majority of it is a proxy. It’s some kind of experimental procedure that we have set up which gives us a window into the biology by making some kind of inferences. If you don’t have a really good understanding of exactly how your assay works, how your experimental process went, you’re really not learning anything about the biology. This is why this metadata question is so important because you can change something in your experimental procedure, it’s going to tell you something different about the biology than the last time you ran it. That’s quite different from a lot of the other datasets that we use in machine learning. Doing image recognition, understanding the source of that image or the date that it was taken doesn’t really change the fact that it’s a cat, doesn’t really change what you’re learning about that image most of the time. Whereas in biology, it’s the whole thing because your assays, as I said, they’re proxies, it’s not the ground truth. That’s why it’s so important to the data teams to have this transparency over exactly how experiments are being run and if you’re looking at two different datasets, you want to combine them together, you need to understand what were the differences in the experimental procedure, what effects do those changes have on the data that comes out the other end. That’s why having this structured metadata is super important, especially if you want to start to really combine a lot of datasets together rather than creating a single dataset for a given experimental question, using it to answer that question and then moving on to the next thing. If you want to say, oh, I’m going to take the last 20 datasets and throw them all at this machine learning model and see if we can learn something fundamental about the biology, you need to really understand what were the differences in those datasets, how were they generated. The question of how to make that experience fluid for the biologist is really the big challenge. The way that it’s tackled now is basically biologists and wet lab biologists will have their own notebooks, they’ll have their own protocols. These are in free text format. They’re not structured at all. If you have a data team, they might sit down with the biologists and say okay which parameters are we going to track, what are we going to put into this big table? These are the things we’re going to track over time. The thing that we want to say is let’s not make that distinction. All of the process that goes into creating a dataset could be important at some point. Can we allow the scientist, when they’re documenting their experiments in the way they do, creating these protocols, creating these steps for themselves, can we do the work using large language models as the principal technology that we’re employing for this particular thing, to allow them to have the flexibility to write things in the way that makes sense to them but then also to take care of a lot of that structuring side on the back end so that when the data scientist comes to look at it, it’s consistent. You can compare different experiments with one another. Does that make sense?
Ross Katz: Yes, it makes sense. I think now would be the right time to ground this in an example. Can you walk me through the step-by-step process for how the metadata gets generated within the experimental design process while interacting with an LLM?
Harry Rickerby: The way that we think about this is we basically went back to the protocol. These are the step-by-step instructions that scientists create or companies create for their scientists to follow when they’re running a given assay or a given experiment. We broke that down into some fundamental concepts that we could structure. We took the information and structured it. The way that a scientist interacts with it right now is in a number of ways. We have that structured hierarchy in our editor. It’s a tool that you can use and you can build this structured version of your method from scratch and you’re defining all of the steps and you’re defining all of the key parameters associated with it. That’s a lot of work. Some of the things that we’re doing to take the burden off the scientist in that process of documenting all of these steps is we’ve built large language models into the tool from the ground up. We think about it a little bit like some of the copilot tools that exist for coders. You can say okay draft me a protocol for X, Y, Z and it’ll give you a good template to start from where a lot of the information’s already structured, a lot of the parameters are defined and then you can tweak it to the specifics of your own experiment. On the other side if you already have something that you’re working from, you already have a plain text protocol somewhere or a description of your method, you can also just paste that into the tool and it will do all of the structuring for you. At the end of it what you get is this clean, easy-to-read, easy-to-understand step-by-step guide for your experiment that you can then integrate more information into without it drowning in the noise. There’s lots of other things that we want to integrate into the tool right now, most principally is your samples. There are tons and tons of different ways that companies handle their samples right now, different tools. We need to integrate with those as well so when the scientist has their structured method we’re also integrating this is where the samples live, here are my different conditions and look after that as well so it’s all included in this structured version of how your experiment went. Does that make sense?
Ross Katz: Yes, it makes sense. It sounds to me like you’re rethinking from the ground up the way that the scientist’s workflow goes, but also trying to leverage the power of these large language models of generative AI to bootstrap a protocol together that has all of the metadata elements and then letting the scientist curate the experimentation protocol that gets generated rather than having to write it all down themselves. You’re getting them to make sure that they fill out all of the elements but you’re also reducing the burden on them to tell you what those elements are. Is that right?
Harry Rickerby: Yeah, exactly. As we go along we want to build in more features that are going to help both the scientist in designing that experiment but also in creating more consistent and more complete documentation. For example, having something which can analyze a protocol and tell you okay maybe there are certain parameters that are missing that you should be including here based on what we’ve seen in the past. Or from the scientist’s perspective, spotting potential errors in their protocol, potential pitfalls or things that look a bit odd that maybe we’ve seen before in the literature. It’s interesting with LLMs. They’re simultaneously incredibly smart, they have consumed so much knowledge. But the specificity of that knowledge to your own experiment is often the problem. How do you make sure that they’re not just going to give you some generic advice that isn’t actually useful for your particular protocol or your particular experiment? We’re thinking a lot about that, how can we give you this copilot that gives you the right kinds of suggestions and the right kinds of tweaks? And also takes care of the structuring and this completeness problem of all of this metadata that we need to capture.
Ross Katz: Another benefit to what you mentioned earlier is improving the reproducibility and the reliability of these biotech experiments. The idea is that this documentation is going to lead scientists to be able to better reproduce experiments that they’ve done in the past or that other people have done in the past. Do you see that playing into this tool and how does the tool inform that process?
Harry Rickerby: Absolutely. This is probably the first reason that we were interested in building Briefly. We thought a lot about reproducibility and the reproducibility crisis broadly in science and we did a lot of digging into why this problem exists, what are some of the root causes of it, both inside companies and the reliability of experiments inside companies but also more broadly in academia as well. The thing that we kept coming back to was so often you don’t have all of the information you need in order to actually reproduce an experiment. We want to give that transparency to an individual or to an organization to say this is exactly how this experiment was run, this is about as much information as we can get. At least this is the intention of the scientist and what they did. There are other things which can be part responsible for problems with reproducibility, but until we have some consistency and have completeness in how experiments are documented, it’s hard to move on to some of those other issues in reproducibility. A lot of the things that we talk about are for example, hidden variables. What was the temperature in the lab that day? What was the lot number of the reagent, maybe it’s gone off. All of those kinds of things are perfectly valid, they’re certainly responsible for a lot of issues with reliability in experimentation, but a lot of what happens when your experiment fails is you think what did I do differently, what did I do wrong? Or what did I do differently from my colleague, or what did I do differently from this paper? The answer to that question is often really hard to figure out. Our first step is okay let’s get to a place where we have a really good record of that and then we can investigate some of these other more hidden kinds of variables and hidden issues that can account for problems with reliability and reproducibility.
Ross Katz: So ideally, everyone would be using a system like Briefly to generate the operating procedure for the experiment and then everybody would be on equal footing in terms of all of the things that get tracked about that experiment so that it would be easier to compare and contrast across experiments.
Harry Rickerby: Right. This ties in with some of the things that we think about a lot which is things like version control is obviously something that’s used in software a huge amount and version control is not something that’s really effectively used in biotech. There’s some kind of document control, especially in regulated biotech environments. But if we’re talking about being able to say okay, show me like GitHub, show me all of the differences in the code between this release and this previous release and highlight all of those differences, it’s really hard to do right now because the information is incomplete, because the information’s inconsistent, it’s not standardized. If you could say okay this data looks funny, I remember it worked last time, show me exactly what changed between today and three months ago. That could help at least answer some of those questions about what could have gone wrong, what could have changed. If things look exactly the same then you can move on to the funny unseen variables.
Ross Katz: So is part of the idea that one scientist could fork the experiment from another scientist and then make some slight changes but that it would be evident in the experiment itself that it’s a fork of this previous experiment? So you could see the experimental lineage of it?
Harry Rickerby: Yeah, definitely. Experimental lineage is super interesting broadly within organizations and also across the scientific landscape. How experiments and protocols evolve and change over years is basically a complete black box. We really just don’t know, a lot of it is hearsay, there are some papers published every now and again about certain methods but a lot of things just crop up to solve specific problems and then maybe hang around forevermore, you never really know why they were there. There’s a good example where there’s a very common media used to grow bacteria. Every single version of this media includes sodium chloride. They did this big experiment where they were like do we actually need this sodium chloride, does it make any difference to how these cells grow? And it doesn’t. This is pretty much as standard as any kind of biological reagent or protocol gets, like LB media. It’s absolutely everywhere and we all put salt in it religiously and it’s actually not doing anything for most of the use cases. Having that transparency about these are where all of these changes came from, now we can understand why they cropped up, what was the reason for them and better understand that ultimately would be super powerful. That would be really awesome. I’m super excited to get to building that.
Ross Katz: So you ask why do we use salt and you get 20 different responses, but the answer is because we always have.
Harry Rickerby: Yeah, pretty much.
Ross Katz: Interesting. What you’re doing with LLMs and building your product on the backbone of LLMs is really fascinating and very innovative inside the biotech space in particular. I want to spend a little bit of time digging in on the way that LLMs are used and the role it plays in your user interface. Can you start by providing how you use LLMs inside the interface? Where the formal software system ends and where the LLM begins?
Harry Rickerby: A lot of the work we’ve done with LLMs in the user interface right now is modeled on, I mentioned before, a lot of the software tools being built for software engineers now are built on the back of LLMs. The underlying thesis behind our use of LLMs is experimental design is fundamentally a creative process. It’s about your instincts, your knowledge as a scientist, your understanding of all of the previous data you’ve generated. All of that stuff feeds into how you’re going to run your next experiment. The thing that we focused on most of all is to say okay LLMs are a really useful tool but what is it that they’re good at and what is it they’re not good at? How do we integrate these into a tool where it doesn’t take away that creative element? It doesn’t take away your ability as a scientist to design the right experiment. It’s not a prescriptive tool. The way that we’ve built them in is to solve specific needs within the context of what we’re building. The first one, and this is the one we started with, is to say okay part of the core of this product is in the structuring of experimental methods. We want to take this mess of how experiments are documented and put them into this more standardized structured format. LLMs are amazing at that. They understand language very deeply, they can convert between different formats extremely well. That’s the first thing that we’ve built in and we’ve built it in a couple of ways. The most basic version of that is just to say okay give me the text that you have, this natural language description of your experiment, paste it into this box and then the LLM will give you the structured version of it which you can read a little bit more easily, that you can follow a bit more easily, it’s nicely structured, nicely laid out. That’s for the scientist and then obviously it’s structured on the back end for when we want to integrate with data teams. Then getting into slightly more interesting use cases where you’re leaning a little bit more on the knowledge that the LLM has. That importing feature, going from free text into the Briefly format is just leaning on the LLM’s amazing abilities with language. But it also has a lot of knowledge. It read a lot of papers, it knows a lot of protocols, it knows how a lot of science is done. The other thing we have is this ability to draft with it. Within the context of the editor you can say okay add a few steps here where I’m washing these cells with PBS. And it’ll give you this structured version of the protocol with those steps drafted for you so you don’t have to go through all of the rigmarole of writing it up yourself. The extent to which that’s good is obviously dependent on how many times the LLM has seen that kind of method before. For very basic methods that it’s seen many times, like a PCR, it can draft you a whole PCR protocol from scratch very reliably. It’s really good at that and it does it really nicely in our tool. If it’s something that’s much more specific and much more niche then obviously it’s not going to do as good a job and so you’re relying more on this interplay between saying oh add a few steps for these particular things here, add a few steps for these particular things here. That’s where we are now in terms of our use of LLMs with the tool. There’s tons more that we can do though. There are so many interesting use cases. But it’s about being really intentional about what problem you’re solving with the LLM. I’ve seen a lot of software products just stick a chatbot into their tool and say hey have a chatbot, you can use it to do whatever. That’s really tempting because LLMs are incredibly pluripotent. They can do so many things, they can answer so many questions. But you’re basically competing with ChatGPT at that point. If you’re building LLMs into a software product you need to be really thoughtful about how it’s actually helping your user do the job that they’re trying to do. That’s our guiding principle behind our use of LLMs and some of the things we’ve been thinking about so far.
Ross Katz: That makes a lot of sense and it’s really insightful. Are you building on the back of one of the APIs, like the OpenAI API?
Harry Rickerby: We’re just using the APIs for now with existing foundational models. In the longer term we’ll probably want to do some fine-tuning and thinking about whether we can train it on specific kinds of data that are particularly relevant for this use case. So far it’s working pretty well with just those foundational models. That’s something we’ll be working on for sure in the future.
Ross Katz: Are you at all concerned about the data privacy issues around the protocol data that you’re sending up and having the LLMs parse? Or do you feel like these protocols are one degree of separation removed from the important data that biotech companies will tend to care about?
Harry Rickerby: It depends on the company. Obviously a lot of companies allow their employees to use ChatGPT. But some don’t. It’s going to be an ongoing question for the field generally, how comfortable do we feel with sending this kind of information to OpenAI or to insert foundational model or insert large language model here. It’s going to be something that we think about a lot. Some companies don’t consider their protocols to be at the level of the data about their molecules for example, but they’re still valuable IP about how they’re doing their science. It’s going to be something we’re super conscious of and we’re thinking about all the time. OpenAI’s working on enterprise versions of these models and obviously a lot of other companies working on it. A lot of interesting stuff is going to be happening in data security with these large language models that we’ll have to keep on top of and figure out how to navigate.
Ross Katz: For sure. Can you talk about some of the challenges that you faced in trying to build your user experience around LLMs? Was there anything you didn’t know going in about building a UX for that that you wish you had known when you started?
Harry Rickerby: It’s a constant learning experience, I would say. The thing that we’re learning most of all with this stuff is really identifying what are the specific use cases where the LLM is going to actually help the user. We have a first version of the product right now where you can just ask the LLM to do things for you in quite a general sense. Generally it’s tough to know when you’re faced with that kind of open expanse of possibility to really know what should I do with this thing, what is it actually going to be useful for? Doing a lot of prototyping, understanding where does it really excel? And then building features around those particular niches is the thing that we’re going to continue to do more of. Otherwise you’re not really offering a huge benefit over just using a chatbot somewhere else. It has to be integrated very deeply with the tool that you’re building. That’s how I’m thinking about it right now.
Ross Katz: So I imagine as you’re experimenting with the tool, you’re finding the limits of where the LLM’s knowledge ends. Are there any unique challenges that you’ve discovered with the highly specialized language that biologists use and the more generalized knowledge that the LLMs have?
Harry Rickerby: I’d say that they’re pretty excellent when it comes to the specialized language of biology. They have an amazing grasp of language and it’s because they have this context around the language that so many other tools in the past have lacked. They will generally have a very good understanding. There are so many acronyms in biology. There are so many acronyms and for me I always struggled personally with that, figuring out remembering what all of these things mean. LLMs are amazing at that because they understand the context around what this acronym is and they’ve seen it a number of times. So the language is less of a problem. Where the problem comes is in the specificity of the experiment that the scientist is running. I find this with a lot of creative tools with AI, one of the problems I often face is say I want to ask an image generation AI to create some picture I have in my head. Wrapping that into a prompt is an impossible task because you have so much context that you’re drawing on, you have all of these ideas in your head about exactly what you want the output to be. Once you’ve coaxed the AI and told it all of those things in order to get exactly the right output you might as well have just created that thing yourself. The same problem applies here and so the thing that we’re not saying very explicitly with what we’re building right now is we can design your experiment for you. I just don’t think we’re really that close to the AI being able to do that because there is just so much context. Maybe if you fed in the results of the last experiment, all of the context that the scientist has, all of the objectives, maybe you could get there but I just don’t think right now it’s realistic. It’s about how do you make the interface as seamless as possible so that the AI can do some of the grunt work and then you can take over and do the specific stuff. Those kinds of things are quite challenging with LLMs. But the language is getting there to be honest and it’s only going to get better.
Ross Katz: These LLMs are so flexible in terms of the outputs that they can give you. I imagine that part of the work of incorporating this really flexible tool into your interface is trying to predict all of the approaches that the scientist might take when designing their experiments so that your system can handle all of the metadata that might come from those different approaches, but also being opinionated enough that you don’t just have this sprawling database of thousands and thousands of columns that have very little data in, trying to simplify the metadata so that you can capture the most important elements. I’m curious if that challenge resonates with you and if so, how do you navigate being opinionated from a software perspective about the data structure versus giving the scientist the flexibility that they need within this very flexible tool, an LLM, to generate the metadata?
Harry Rickerby: When we started building this tool we did a lot of research, basically just read as many experimental designs, as many protocols, as many methods as we could get our hands on and tried to break it down into some meaningful concepts that make sense both from the scientist’s perspective but also are useful when you lay them out in a certain way that will then be useful when you want to query that data later. That formed the basis of the schema that we have for describing how a protocol should be documented, what are all of these different things. We have that as this ground level and it’s very broad. It’s not specific particularly to an area of science or even science in general, it’s more of a description of a process but it’s with science in mind and with some of the concepts of scientific experiments in mind. That’s where we’ve been opinionated, in that schema. Where we’re not being opinionated is in what actually goes into that schema. When the scientist decides okay I’m going to document my experiment and for example, I want to spin my cells down, we’re not going to say you have to write centrifuge. We’re not going to force a certain language or a certain way of describing an experiment onto the scientist while they’re designing the experiment. What our hypothesis is and the idea going forward is to say okay on the back end that’s something that we will deal with because we’re now in a place where we have tools which can automate the standardization and understanding of language that we didn’t have three years ago. We can start to group and cluster all of these different concepts together and start to standardize them on the back end so that when you want to query a database of all of this information you can do so in a meaningful way. You don’t have to deal with all of that flexibility, it’s dealt with for you in the data structure and the way that we cluster things. The language itself is something we want to leave flexible because otherwise the tool just becomes overbearing. It restricts the ways that scientists will express a lot of these things and that can become really problematic.
Ross Katz: What you just said is super interesting to me. Rather than forcing the scientist to apply a tag or to use the term centrifuge when talking about spinning the cells, you’re planning to do post-processing steps that apply that tag on the back end so that when you then want to query your data about all of the experiments that have centrifuging in them, that’s something that you can do within the existing data model without having to force the scientist to do that up front. Is that right?
Harry Rickerby: Yeah, that’s exactly it.
Ross Katz: Interesting. I know that you’re a startup, so I assume there’s a plan at some point to sell the software system into biotech R&D organizations. How do you think about the challenges of bringing a system like this to market, especially when you’re so focused on the needs of the individual scientist?
Harry Rickerby: It’s obviously a question that we talk about a lot. Having this inconsistency and incompleteness in how all of these experiments are documented and all of the problems that has for data teams, all of those problems resonate with a lot of companies. The key for us is focusing on how do we make sure that the tool is really useful for scientists, that they’re going to use it, that the information that gets into the system is actually much better than it was before. If we can demonstrate that and if scientists like using it and it’s useful for them, then at that point we’d say okay great, let’s see if we can start to implement this for an organization as a whole. For right now it’s about making sure that we’re really nailing the needs of the scientist and making sure that the tool is genuinely useful for them as an individual and then building all of that collaboration, building all of that data on and selling into those companies at that point should be much easier. That’s our approach right now.
Ross Katz: If there’s a scientist interested in working with the tool or a company interested in prototyping the tool internally, what’s the best way for people to learn more about you and the work you’re doing at Briefly?
Harry Rickerby: Our website is briefly.bio. You can find us also on LinkedIn. I have a Substack, Harry Rickerby, where I’ve been writing about some of these things myself, if you want to follow. If you’re a scientist, you can sign up to start prototyping the tool. We’re in early access now. If you go to briefly.bio/builder you can join the waitlist and we’ll get you into the tool so you can start playing with it and seeing if it’s going to be useful for you.
Ross Katz: Awesome. Just as we come to a close, are there any pieces of content, things that you’ve read or listened to recently that you think would be valuable for people in your arena to encounter if they want to learn more about this type of work?
Harry Rickerby: The blog that I’ve followed for a while and has been talking about this space for a really long time is Jesse Johnson’s blog, Scaling Biotech. He approaches it from working inside a biotech company, building a data team, figuring out the answers to all of these questions and how to build your systems and your team structure and your culture to tackle all of this. It’s a really great resource. I would recommend checking that out if you’re interested in this kind of stuff.
Ross Katz: The podcast episode with Jesse should be out by the time this episode airs. For anyone who is interested in learning more about Jesse’s perspectives, just scroll down in the feed and you’ll definitely get to hear Jesse’s thoughts on this arena.
Harry Rickerby: He’s been doing it way longer than I have, so I would definitely listen to him first.
Ross Katz: Harry, this has been a super fascinating conversation. I really enjoyed hearing what you’re doing at Briefly and excited to see what’s in store for you all in the months and years to come. Thanks for joining us.
Harry Rickerby: Thank you so much, Ross. It was great fun. Thanks a lot.
Jason: And that’s it for this episode of Data in Biotech. If you enjoyed the episode, please subscribe, rate, or leave a review in your podcast platform of choice. See you next time.
