Listen on
Overview
A senior lead on the manufacturing floor spends four to six hours reconstructing what happened on a million-dollar oligo synthesizer when a run goes sideways. Engineers, process teams, and operators each hold a slice of the answer, and they often do not trust each other enough to share it cleanly. Meanwhile, sensor data on pressures, temperatures, valve states, and vacuum systems is piling up faster than anyone can query it, and the operations team has no way to tell an anomalous event apart from regular day-to-day variation.
This is a special in-house episode. Host Ross Katz talks with James Winegar, CEO of CorrDyn, about the firm’s multi-year work with a custom oligo manufacturer. James walks through the system CorrDyn built to process tens of terabytes per month of synthesizer time-series data, deliver answers inside two to five minutes, and surface them through Grafana dashboards that cut senior-lead investigations from four to six hours down to five minutes. He explains why near-real-time at a tenth of the cost beat sub-second real-time for this client, how visual inspection ML changes in-situ quality control for downstream lot decisions, and why trust gaps between data science and operations teams cause otherwise sound projects to stall. The conversation closes on scoping: reframing a color-classification problem as a binary detection task to capture ninety to ninety-five percent of the value at a tenth of the work.
Key Takeaways
Near-real-time at a tenth of the cost beats sub-second real-time for most manufacturing QC
When CorrDyn scoped the oligo manufacturer’s system, the client chose two-to-five-minute query latency over sub-second latency because it cost roughly ten times less and still cleared the bar for every active use case. The work that used to take a senior lead four to six hours now takes five minutes, which is the number that matters for throughput and for freeing up scarce expertise.
Communication scales quadratically, so external facilitators often unblock projects faster than more internal meetings
In a hundred-person organization, roughly a thousand pairwise touchpoints have to happen for everyone to stay aligned, and teams rarely create healthy space for that discourse. A third party who can sit with IT, security, and manufacturing and extract the real underlying constraints can short-circuit months of back-and-forth, particularly when the blocker is stonewalling between operations and data science over who owns a production ML system.
Reframing the objective is usually worth more than improving the model
One client needed to classify the color of material in a well — white, blue, black, or orange — which is a non-trivial computer vision problem. Reframing it as “is there a thing in the well or not” solved ninety to ninety-five percent of the business problem at roughly a tenth of the effort. Chasing the last increment of accuracy almost always destroys the ROI case, and the job of the senior technical stakeholder is to surface that tradeoff before the work starts.
Related: CorrDyn works with biotech and life sciences manufacturers on exactly this class of problem. See our data engineering practice for streaming systems and machine learning for production-grade QC models.
Full Transcript
Jason: Welcome to Data in Biotech, a podcast from CorrDyn where we explore how companies leverage data to drive innovation in life sciences. Every two weeks, we sit down with an expert from the world of biotechnology to understand how they’re using data science to solve technical challenges, streamline operations, and further innovation in their business. Today, we’re doing something a little different. We spoke to James Winegar, CEO of CorrDyn and the company that produces this podcast. CorrDyn works with some of the largest biotech manufacturers in the US, helping these companies make smarter strategic decisions to unlocking the value of their data. In this conversation, James and Ross discuss the application of machine learning in biotech manufacturing quality control, their work in providing a custom oligo manufacturer with a data analysis system to track whether certain manufacturing events are anomalous or part of regular operations, and how trust issues between internal teams can lead to conflict, misunderstandings, and ultimately, failure to deliver ROI. Here we go.
Ross Katz: I know that you’ve worked with several biotech organizations in the past. Can you give us an example of the kinds of projects that you’ve done or of a project that you’ve done that has been particularly valuable to the client that you’ve served?
James Winegar: We have a client that manufactures custom oligos and we’ve done dozens of statements of work with them and that’s ranged the whole gamut of data systems, infrastructure work, embedded systems, etc. but what was most valuable to them from us is setting them up with a system that they were able to answer questions effectively and quickly. When we first started working with them, we did this assessment and during that assessment, we were actually put up against a few competitors. We came out on top because they thought that we really understood what they were trying to accomplish from a functional viewpoint, not just from a technical viewpoint. Based off of that understanding, we put together a really robust system for analyzing their data in reasonably close to real time. The question we asked them was how much does real time actually matter versus the cost of the system. They said if we can answer questions within a couple minutes, that’s a tenth of the cost, that’s better than being able to answer questions within seconds because right now a lot of things are currently manual. So what can we do to enable people to make questions quickly in minutes instead of hours. We put together this system. It’s a streaming system that’s relatively low cost for them. We process tens of terabytes of data every month and from there, they’re able to answer most of their questions within two to five minutes and that’s good enough to support the vast majority of use cases that they have. In doing that, our biggest impact’s been on improving the efficiency of the key people within the organization. What used to take one of the senior leads in handling some of these issues, what took them hours, four to six hours, is now taking five minutes because we have the system set up. Real time data goes in. It ends up in a Grafana dashboard and helped them understand what’s happening on the machine over time. They have a decent amount of history to look back at, understand what’s going on and I think it’s been transformative to the way that their operations run on the manufacturing side, as well as enabling the engineering organization to make more effective project plans about what work they should be focused on because they know what’s the impact of some of the things that they’re doing on a given machine from waste management or throughput. There’s a multitude sets of dimensions that they can work on.
Ross Katz: Right. This client is a manufacturer, so can you give us a sense of what are the types of data that are being collected and analyzed in the system and what are the sorts of questions that people are able to answer more efficiently based on that data?
James Winegar: This is a custom oligo manufacturer. They’re synthesizing oligos. That’s where the vast majority of the data we are supporting them on — we support them in other areas, but the synthesizers are the most important. All a synthesizer is is a machine and it’s going to push around different reagents. It’s going to open and close valves. It’s going to have a vacuum system, etc. and all that information is just a bunch of time series data. It’s pressure over time, temperature over time, etc. for each and every sensor that’s existing within this device. That creates a pretty complicated data set. But what it allows us to do is to rationalize that data using a tool like Grafana.
Ross Katz: When you say rationalize, what do you mean by rationalize?
James Winegar: On this machine, it’s very complex machine, let’s just say it’s a million dollar built out cost. There’s multiple things happening at the same time or within a small time window. What we want to do is figure out how these different interactions that the machine is having are impacting the overall manufacturing quality or is this an anomalous event or is this regular day-to-day business and we don’t have to care about it? Then from there, instead of having somebody have to figure it out every single time, we can create at least some basic rules to enable whatever operator to understand hey there’s a situation going on on machine A. Let’s go look at machine A and figure out whether or not we should take it offline for maintenance or if we should let it continue running for another twelve hours until we think it’s ready to go.
Ross Katz: That makes sense. Who are the types of stakeholder groups that are consuming this data and what are they doing with it?
James Winegar: It’s a lot of different groups at this client. There’s engineering themselves because they have an interest in improving the quality of the overall machine, what changes should they make to the underlying architecture of the synthesizer. We have process engineers, they’re interested in understanding should they change synthesis flow, etc. through this system so that they reduce failures, improve throughput, etc. Then there’s also just the regular day-to-day operators that are on the floor. There’s the maintenance team as well as your regular operator and historically they had a lot of conflict between each other because they just didn’t know who was responsible for handling a situation at a given time. Another thing that the work that we’ve done for them has enabled is for those teams instead of being confrontational, to partner with each other and help each other make good decisions.
Ross Katz: Right. Vast quantities of sensor data from the machines are being collected in near real time and they’re being pushed through this system and then they’re being consumed by these various stakeholder groups to answer the questions that are most critical to ensuring the quality of production, ensuring that production is moving through on time. But that use case is focused on how humans use the data. Can you give some examples of where you’ve taken the next step to use machine learning to help make decisions based on the data?
James Winegar: We have a lot of different use cases for our customers where they have visual inspection that happens on a plate or some device or really anything. The advances in machine learning against image use cases has improved dramatically over the last five years. The common problem statement we get is I want to identify a failure in situ at point of time on device because there’s downstream consequences of not ending a failed product that I can potentially know is failed during the manufacturing cycle. Lost revenue, lost time, all the stuff really ties back to money at the end of the day and efficiency and throughput but if you have FDA requirements about lots, you might want to throw out things that you definitely know are going to fail during your subselection to determine whether or not the lot has to be scrapped. You could have just cost of maybe my QC is hours or days away from my actual manufacturing time and there’s just a lot of lost opportunity in those scenarios. Just in terms of some examples that we can talk about, we have a device that’s coming through and applying a bunch of heat to some plastic and you can look at the plastic and say it’s not connected appropriately, so that’s a failure. You have to collect a bunch of data. A lot of times the problem becomes getting the information for the customer. How do I collect these images of an in situ process, can I attach a camera, is the orientation of that camera going to change? There’s a lot of things that can happen on a production floor. Let’s just say I thought through all that stuff and I feel comfortable with it, I take a bunch of pictures. I got ten thousand, fifteen thousand images of whatever device I’m working with. Then somebody has to go through and look at them and say whether or not it’s good or not. That takes a lot of labor. Honestly, when we’re helping our customers, a lot of our effort is to help them get that process nailed down to be as efficiently as possible so that they can actually make this jump into in situ quality controls because without that then they have that downstream effect that they have to worry about.
Ross Katz: Why do these clients with these sorts of quality control and manufacturing problems bring CorrDyn in to solve them? Are there skills gaps at the organizations or is there some sort of disparate understanding between the different groups that would need to collaborate together? Why hire externally for this sort of challenge?
James Winegar: I think that bringing in a team of people who have solved a problem before can short circuit some of the issues you would have on a project like this internally. A lot of our customers are capable of doing this themselves theoretically, but having us around to support them helps short circuit potentially months of back and forth because I can talk with IT and security and manufacturing and figure out here’s your actual concerns that you care about which are really independent of us being a third party. We just want them to get value from the system we build, but teams might have competing priorities against each other and then we can be the tiebreaker to be like here’s what makes sense given the overlap of your constraints and how we can meet the actual fundamental requirements. Getting people to talk about what is the real thing that they’re trying to get at is honestly half of the battle for a lot of these internal teams because they don’t trust each other for various reasons.
Ross Katz: You mentioned that there are sometimes trust issues between these different teams that require an external expert facilitator to come in and help resolve the conflict so that the big important job can get done. Why do these sorts of conflicts exist do you think?
James Winegar: I think that it’s just human nature. Any sufficiently large organization or group of people is going to have conflicts between each other. Communication scales quadratically with people. When I think about a hundred person organization, that means there’s a thousand ish touchpoints that have to happen for everybody to be on the same page. Sometimes people don’t create the space for those touchpoints to exist in a way that’s healthy for discourse.
Ross Katz: I imagine that each project that gets worked on doesn’t exist in a vacuum. There’s this long interaction pattern between these different teams that are bigger than the individuals within the teams, but that causes communication and coming to agreement on big projects like this to be challenging. Can you give any examples of how you’ve seen this lack of trust result in negative consequences for an organization? What’s the outcome of lack of trust?
James Winegar: The big outcome of lack of trust is stonewalling against a project. I think an example is actually better here. There’s a client of ours, there’s a data science team and then there’s an operations team. The operations team could really use what the data science team wants to bring to the table. However, that comes with a significant amount of process overhead that they now have to worry about in addition to what they already worry about. Who maintains that, who continues to make sure that it’s effective, are all the processes in place to make sure that the operations team doesn’t have this burden applied to them that was just a POC by a data science team? Is it actually going to be managed effectively with the skills that the operations team has, which they’re not going to hire a data scientist on staff typically? Or really somebody who can handle a complex system like a machine learning inferencing use case. That’s really where I think the distrust is because they don’t have ownership across the boundaries of the teams for what’s going to be delivered. How do you set up systems, how do you set up processes to support those multidisciplinary teams? And make it reasonably easy for people to do what they need to do. There’s always some challenge there because technology has a shiny tool problem, but if you sit back and you go what tools exist to help solve this problem that the operations team is going to actually be able to support? How do I give them the capability to update the labels on new images coming across the line where they had defects that weren’t actually labeled defects that weren’t actually defects and then every day, week, month, whatever, retrains their model, redeploys it and they’re just stewards of the system at that point and they don’t have to have ownership over a deep learning algorithm to support this use case, which might have very high revenue implications for them.
Jason: Are you a biotechnology company looking to unlock the potential of your business data? CorrDyn can help. We’re an enterprise data specialist that helps companies working in life sciences make smarter strategic decisions. From developing the right data strategy that starts with our data maturity assessment to building and delivering bespoke technical solutions, we are equipped to tackle the most complex data challenges. We have partnered with dozens of high growth organizations, from manufacturers of custom oligonucleotides to molecular diagnostic companies to achieve data competency. Whether you need to supplement existing technology teams with specialist expertise or launch a data program that lays the groundwork for future internal hires, you can partner with CorrDyn to unlock the potential of your business data today. Simply visit connect.corrdyn.com/biotech to learn more. Now back to the show.
Ross Katz: There’s a fear of being forced to take on more work that is outside the scope of how each of these stakeholder groups views their role. Part of the planning process for a machine learning project in a biotech organization is making sure that there’s a sustainability plan so that there’s an appropriate division of labor between who owns which portions of the system, but that the feedback loops that you need to create between the knowledge that’s in people’s heads and the knowledge that you’re embedding in a machine learning model for example gets translated, gets transferred over. Am I thinking about that right?
James Winegar: I’m going to reframe it a little bit, but when I think about machine learning projects in general, I think about the R&D side and you can map that directly over to the R&D side with a biotech organization as well and the production side. The concerns in production are very different. It’s more about reliability, maintainability of the system, whereas on the R&D side, it’s that academic versus professional environment again because the R&D side’s still academic in their approach to a lot of things. A lot of data science projects or machine learning projects are coming at things from that academic viewpoint. This is actually where that skill set gap can come in — many of the people working there haven’t actually had to productionalize and take ownership of a system that went out to the manufacturing floor. Some of it is also the communication side of things, how do we help people understand that the operations team is not actually going to be able to own this. They might have the labor that’s required for the labeling and will be stewards of the system, but if something needs to be adjusted in the model, the operations team’s not going to do that. That needs to come back to the data science team and understanding what the implications are, helping the stakeholder walk through the journey is a major part of what we do.
Ross Katz: Interesting. From a leadership perspective, just having all of that context laid out for you about what the staffing plan needs to look like, how people’s roles are going to change in owning a system like this, are there other elements that you feel like leaders of biotech organizations need to be aware of as they’re thinking about bringing a machine learning model into production?
James Winegar: Minimization of complexity is one of the most important things you can do in a system. If you are trying to solve a particular problem, is it possible to reframe the problem statement in a more tangible objective and then focus the machine learning process against that instead? Maybe that’s from a data collection or a labeling standpoint. It’s just easier to understand what’s happening. There might be some tradeoffs for that, but then helping that stakeholder understand these tradeoffs and make appropriate decisions while continuing to make forward progress is really important. These are the types of projects where you can have delays because you can’t get equipment installed for months if you don’t have proper alignment going into the project.
Ross Katz: You mentioned simplifying the objective that the model’s trying to measure, is there an example of what that simplification looks like in practice?
James Winegar: I’ll just use an example from one of our customers. We have a plate that’s got however many wells on that plate and the question that was phrased was what color is the thing in the well? Is it white, is it blue, is it black, is it orange? It’s a color statement. Well that’s actually pretty hard because the way cameras work and things like that. In talking with them, we’re like how much value is there if I say there’s a thing or not? Binary decision making instead of what is the color, which requires a pretty non-trivial approach to things. And they were like that doesn’t solve all of my problems, but it solves ninety to ninety-five percent of my problems and it’s like I can do ninety to ninety-five percent of your problems in probably a tenth of the work. Would you like us to work on the ninety to ninety-five percent solution or do you want to do the hundred percent solution? Our stakeholder there is quite pragmatic and so he said ninety to ninety-five percent is exactly what we’re looking for here.
Ross Katz: Having experience with lots of machine learning problems and putting them into production allows you to analyze the problem statement and determine whether the problem statement is more or less achievable based on the approaches that are out there. Is that right?
James Winegar: That’s exactly what I’m thinking in terms of you can do as many complicated things as you want and maybe get some better result. Is that incremental gain actually going to be worth it to what needs to be accomplished? The time spent for that incremental gain will completely blow out the ROI against what actually needs to be solved for.
Ross Katz: That gets back to the conversations that you’re having with the leadership of these sorts of data initiatives. It’s giving them the context to make smart decisions about which problems get worked on based on the ROI that those problems can achieve.
James Winegar: I think it’s a few different dimensions. ROI is almost always the biggest driver, but achievability or time to value are the other drivers. I can do this in a month or we can spend a year on a project. A lot of people will say sooner all the time, but if you’re talking with a good stakeholder, they’ll be like if we do it sooner we’ll never do it right and the implications of doing it right will have a larger impact on the long tail of the ROI. I think we should invest in that long tail. Understanding what the implications are, helping the stakeholder walk through the journey is a major part of what we do. Some people might respond to that saying you have a plan to do something now, you can get that out and then work on this program that gives you more and we can justify that continued investment in it because that delta is good enough, but other people might just say no do the thing correctly the first time because we don’t have competitive pressure to put it out prior to your timeline but you need to make sure you deliver against this timeline that you have outlined. The other way might just be get something out there and then we’re going to leave it forever and it’ll exist for twenty years and we’ll just capture what we can now. It’s culture and who’s the budget owner, stakeholder that’s going to be responsible for the dollars and cents at the end of the day and how they are positioning themselves.
Ross Katz: That was what came to mind for me too as you were talking. It sounds like the context in which the biotech leader is operating in in terms of who their stakeholders are and what their expectations are in terms of the timeline in which value needs to be generated. Also the level of trust that those stakeholders have in the person who’s taking ownership of a data project like this to go on a multi-year process before the value is really demonstrated versus needing to see immediate results in order to get there.
James Winegar: Exactly. A lot of people are pressured to get results as quickly as possible because that’s presentable at that point. Instead of having to continue to flush money down the toilet.
Ross Katz: As we draw this conversation to a close, I just want to give you an opportunity to share are there any other ideas that occurred to you as we were talking that are things that you wanted to make sure to share with data stakeholders who are in biotech?
James Winegar: I want everybody to really evaluate what they’re trying to accomplish up front. If you spend the time to think about what do you really want to accomplish, you can really improve the overall process in which you go through instead of having to pivot a bunch of times. Honestly that’s just putting together a structured workshop with key stakeholders, making sure it’s not too big and enabling people to get their ideas out and then letting somebody work on the problem statement and what are we trying to accomplish, why are we trying to accomplish it, what’s the impact of accomplishing that maybe from a functional viewpoint, maybe from a financial viewpoint. It really depends on what people are trying to drive at. Once you have that in place, it’s really much easier to deliver on a project. There’s a lot of projects we get called about where the problem statement isn’t really well defined and we always start with what are we trying to really get at here. When we help answer that question, I think that’s where we’re producing a lot of value.
Ross Katz: James, really appreciate you joining the podcast today and look forward to continuing the conversation at a later date.
James Winegar: Thanks.
Jason: And that’s it for this episode of Data in Biotech. If you enjoyed the episode, please subscribe, rate, or leave a review in your podcast platform of choice. See you next time.
