Listen on
Overview
Biomedical AI has become very good at demonstrating capability and is still bad at earning clinical trust. A model can post strong numbers on a benchmark and still fail to behave consistently in the high-stakes setting where a clinician would actually rely on it. Arvind Rao argues the field’s central failure is not algorithmic but translational: the gap between a model that performs well in a paper and one that can be trusted, run after run, on a real patient.
Arvind Rao is Professor of Computational Medicine and Bioinformatics, with a joint appointment in Radiation Oncology, at the University of Michigan. His lab works across H&E histopathology, multiplex immunofluorescence, spatial transcriptomics, and single-cell RNA sequencing — not only to build predictive models but to trace the full lifecycle from data to model to inference and ask where that lifecycle can be trusted and where it cannot. Alongside the research, he builds AI literacy programs that help clinical and procurement teams evaluate and govern the AI systems vendors are selling them.
In this episode, host Ross Katz and Arvind work through two of his recent papers — SPIFEE, a graph-based framework that replaces single-number interaction scores in the tumor microenvironment with spatially resolved representations, and a multimodal framework that traces a path from a stained tissue slide to a nominated drug target — then step back to why Goodhart’s Law applies to foundation model evaluation in biology, why generative AI’s variability makes it unsuitable for clinical deployment without credentialing, and what AI literacy actually means for the people signing the purchase orders.
Key Takeaways
The hard problem in biomedical AI is translation, not algorithms
The field is good at capability demonstration: a model that segments a tumor, predicts response, or reads a slide well enough to publish. It is far weaker at the next step — showing that the same model behaves consistently enough to be trusted in a clinical decision. Arvind frames this as the gap between demonstrating a capability once and deploying it where a wrong answer carries a cost. Closing that gap is less about a better architecture than about characterizing how a model behaves across the full data-to-decision pipeline.
The tumor microenvironment is not a number
Conventional analysis of the tumor microenvironment collapses cell-cell interactions into scalar scores — a single number for how much two cell types co-occur. SPIFEE, the graph-based framework Arvind’s lab built, replaces that scalar edge with a spatially resolved functional representation: not just whether two cell types mix, but where and in what pattern. That structure separates disease states the single-number approach could not — chronic inflammation from pancreatic cancer, an indolent lesion from an aggressive one.
A path from stained tissue to a nominated drug target
His multimodal framework starts from H&E pathology slides labeled only with clinical outcomes, uses multiple instance learning to discover the morphological patterns that track those outcomes, maps those patterns onto spatial transcriptomics to recover the molecular signal underneath, and ends by nominating mechanisms and actionable drug targets. The point is the chain: each modality hands off to the next, so a visual pattern on a slide becomes a hypothesis about biology a researcher can act on.
Goodhart’s Law comes for foundation model evaluation
When a measure becomes a target, it stops being a good measure. Arvind argues that applies directly to foundation models in biology: optimize hard against a benchmark and you get models that score well on the benchmark, not necessarily models that understand the underlying biology. The benchmark and the goal drift apart exactly when the stakes are highest, which is why he treats evaluation design as a first-class problem rather than a final checkbox.
Variability is a feature of generative AI and a liability in the clinic
Generative models sample from a distribution, so they produce different outputs on identical inputs by design. Chain several generative agents into a clinical workflow and that variability compounds across steps. The frameworks that would let you bound and characterize it — the equivalent of calibrating a pipette or validating a diagnostic device — largely do not exist yet for these systems. Until they do, capability demonstration and clinical deployment remain two different things.
AI literacy is a procurement skill, not just an engineering one
When clinical and procurement teams cannot evaluate a vendor’s claims, they buy on demos. Arvind’s AI literacy work gives non-specialists the sniff-test questions to ask before trusting a system: what it was evaluated on, how it behaves when it is wrong, and what happens at the edges of its intended use. The literacy gap is expensive precisely because its cost is hidden — you only see it once a system you could not interrogate is already in the workflow.
Related: CorrDyn partners with biotech, healthcare, and life sciences organizations on AI strategy, machine learning, and the evaluation and governance decisions that determine whether a model can be trusted in production.
Full Transcript
Jason: Welcome to Data in Biotech, a podcast from CorrDyn where we explore how companies leverage data to drive innovation in life sciences. Every two weeks we sit down with an expert from the world of biotechnology to understand how they’re using data science to solve technical challenges, streamline operations, and further innovation in their business. Here we go.
Ross Katz: Welcome to Data in Biotech, I’m Ross Katz, Data Science Lead at CorrDyn. There’s a growing assumption in biotech that if you throw enough omics layers into a model, clinical breakthroughs will follow, but the reality is that integrating multimodal biological data is full of pitfalls that can quietly compromise the quality of your predictions and your science. Arvind Rao is a Professor of Computational Medicine and Bioinformatics at the University of Michigan, where he leads work on merging multi-omics datasets for cancer research and has built AI literacy programs designed to help healthcare and pharma teams to better evaluate AI tools on the market. Arvind, welcome to the show.
Arvind Rao: Hey Ross, thanks for having me.
Ross Katz: To get us started, you sit at the intersection of computational biology, clinical oncology, and AI education. Can you give us a sense of the landscape you operate in and what your day-to-day research looks like right now?
Arvind Rao: Sure. What me and my group tend to be doing most of the time is trying to answer a really fundamental question, which is how can we set trust guidelines in the context of biomedical AI? And this becomes a very multi-dimensional question because it’s not just about building the right models with the right kind of data, but it’s also about taking the outputs of those models to then biologically credible or clinically credible inference. That’s really the question where we start: how can we actually trust biomedical AI algorithms? And that leads us to working across pathology images, spatial transcriptomics, bulk RNA-seq, single-cell data, not just to build models but to understand what information is real, what’s recoverable fundamentally, and what’s just noise dressed up as signal. We also look a lot at reliability, credentialing, and model veracity. And as I’ll say a little bit more as we go on, we’re trying to understand the complete life cycle that goes from data to model to inference and probably even feedback from each of those three different spouts back to how we do things better. That’s one part of what we do. In parallel, I also work with the pharma and healthcare teams to help them evaluate these systems critically, because the bottleneck right now is not the models, it’s in the decision making under uncertainty. Because not all modalities are available simultaneously all the time for every single patient, a lot of these decisions have to be made when full information is not available, and because the calibration of those sources feeds into how one would meaningfully make decisions, that’s another thing that we work on quite a bit.
Ross Katz: Touching on the kinds of data that you work with and how you combine them together — we’re talking about multimodal and multi-omics data integration for cancer, and your recent work spans H&E histopathology, multiplex immunofluorescence, spatial transcriptomics, single-cell RNA sequences, as you were just talking about. For listeners who may know some of these but not all, can you briefly touch on what each of these data types show and what the integration problems are when you try to combine them?
Arvind Rao: Sure. I’ll step back for just a little bit to communicate to the audience who may not be from biology or clinical sciences how unprecedented a time we are living in, which is that biology has become extremely quantitative. We have access to a lot of modalities, a lot of technologies that let us go all the way from tissue organization to cells to subcellular and information about proteins and genes and all these different constituents. It’s really a multi-scale, multi-phenotype, multi-omic problem that has become the lingua franca of biology. Which means that the ability to understand these disparate yet correlated views, making sense of this data, trying to understand the biological or clinical picture, the pathology, is all becoming extremely quantitatively grounded. It’s a great time to be a data scientist in biology and clinical sciences. With that said, each modality gives you a somewhat different view of the phenomenon. We work in oncology, so I’m talking about things in the context of cancer. Every modality gives you a different view of the tumor. H&E images, which have been around for more than a hundred years, show structural and morphological aspects of disease. How are the cells, are they atypical, are they weirdly shaped, etc. We can get that good sense from the H&E images. And we can now take pictures of those images as well, digitized, and you can do lots of interesting image processing with them. Single-cell RNA-seq is a more recent development that gives you a look into these different cells and their states at very high resolution. You can get close to anywhere from five thousand to ten thousand genes and their expression recovered from every single cell in your tissue. Spatial transcriptomics takes that one step further and says not only will it tell you what’s in the cell, it will also be able to orient you in terms of where that cell is located relative to the pathology of the disease. You can actually get a sense for which cells come together, which cells tend to be separated, and whether there are histological manifestations within which these cells tend to express themselves or be prevalent. A somewhat lower cost variation of that is multiplex imaging, where you may not want to get information about all the genes or all the proteins, but some selected few. If you know what you’re looking for, multiplex imaging becomes a very good way to understand which cell types are present and how they are oriented in the microenvironment. On challenges: because these are overlapping yet disparate views of the tumor, I almost liken it to remote sensing or satellite imagery. You can get hyperspectral images of very few areas in deep resolution, excellent spatial resolution, excellent wavelength fidelity, but that’s very expensive. You can’t do that all the time. If you do the traditional multispectral or low-res satellite imaging, you can get a lot of data, but not at that resolution. The ability to meld these two different scales of data to reconstruct the biology becomes really critical. One challenge is these images don’t align naturally. One has to look for coordinate-free approaches to be able to put these together. Second, despite these images being fantastic and unprecedented in terms of insight, they can differ quite a bit in resolution, noise characteristics, sampling biases, batch effects, and even in what signal means relative to noise. It’s not just a simple matter of integration — it’s not simply stacking data, it’s reconciling these fundamentally different representations of biology, and that’s where the effort is currently oriented.
Ross Katz: I’d love to dive into some examples of your work to illustrate how you’re using these different modalities. There are two recent papers that you’ve shared with us, both published this year in NPJ Artificial Intelligence. I think they tell a compelling story about how you do these coordinate-free approaches to data integration that surface biological context and allow you to draw verifiable conclusions about what’s going on. If it’s okay with you, I’d like to start with SPIFEE. My understanding is that most graph-based approaches to looking at the tumor microenvironment represent the relationship between two cell types as a single number, a distance or an interaction score. And SPIFEE replaces that with an entire function. Can you explain what information you integrated into that, what we were losing before, and what SPIFEE allows you to capture and utilize that you couldn’t previously?
Arvind Rao: That was a very good articulation of the paper. Before I get into SPIFEE, I have to give a shout out to the student Ashley Tsang, who did the work in leading us on this paper. It was really her hard work that brought it all together. The question we sought to ask was exactly what you said: given the exquisite spatial resolution of these platforms — revealing not just what cell types are there in the environment but how they are oriented relative to each other — the prior technology was mostly summarizing all of that complexity with a single number. Which meant that if things are heavily infiltrated, heavily mixed, you were summarizing it to a single number because it made the downstream analysis and statistics easier. But we felt it was too reductive in the sense that there are patterns to these spatial infiltrations. Two cell types might be mixing with each other, but the extent to which they mix might have a spatial component, which means it might not be uniformly mixing — it might be mixing in certain pockets but not everywhere. That’s a very different picture than uniform mixing. Unless you can go beyond a single number that summarizes all of that complexity, you’re going to be losing out on the insights you could get. With that in mind, Ashley and I set out to ask: what if we used a measurement that looks at not just the amount of mixing but where that mixing happens? And can we express that as a function in space? What we came up with was an abstraction where we had a graph, so we could leverage the graph neural network paradigm, but the graph was now not just simply nodes and edges — both of which were scalar — but the nodes could be vectors, which gets at the complexity of the cell identity, and the edges could be functions, which gave us this notion of what is the spatial pattern of intermixing between any two of these cell types. You could express that very compactly with a graph, but now adding this functionally valued edge on top. It did take us some work to repurpose graph neural networks from scalars or even vectors to functions, because it’s not a trivial jump. But thanks to Ashley’s persistence and hard work, we were able to get to a point where these objects can now be analyzed en masse and we can get a much better sense of spatial infiltration between tumor cells and different immune cells, both statically and dynamically over space, and use that as a way to get signatures of how these relationships change and whether those changes in relationships can be potentially prognostic or predictive. That was the goal we sought with the SPIFEE paper.
Ross Katz: Can you give an example of where that kind of intermixing logic exists and the types of conclusions it allows you to draw?
Arvind Rao: Practically anything about cancer is about the microenvironment. Everybody in the field agrees the fundamental hallmarks of cancer involve immune modulation, which means you’ve got this plethora of immune cell types in the microenvironment which get recruited in very interesting ways when there is this fight between the invading tumor and the resisting host. Each one’s trying to do their thing and this push and pull mechanism creates a very dynamic picture of how these immune cells are oriented relative to not just each other but also to the tumor. Pancreatic cancer is one example that we used in the paper, but anything to do with microenvironment automatically falls into this scope. One can then ask how these spatial differences, when we use this kind of SPIFEE approach, reveal insights into the biology of the disease that you couldn’t have gotten otherwise. It’s not just the interaction, it’s the structure of the interaction in the microenvironment, which can distinguish different states. In the case of pancreatic cancer, one can for example distinguish chronic inflammation from cancer, or an intraductal papillary mucinous neoplasm from a more aggressive cancer. This has implications for understanding the stage of disease in ways that we were unable to find elsewhere.
Ross Katz: I found the paper really interesting — the way you’re constructing the models allows you to identify not just what is happening or what the predictive outcome is, but where it’s happening and then dig into the how. This is a beautiful marriage of deep biological understanding and algorithmic approaches that allow you to get to where to focus your attention much faster. From a high-level healthcare lens, it’s using computationally efficient approaches that allow you to focus the most limited resource of all — expert attention — on the problem that might be going on with a given patient. Am I thinking about that right?
Arvind Rao: Absolutely. It speaks to the notion of interpretability, which becomes part of the conversation. You now have the ability, because you can also get images that are stacked in space, to say: this part goes for immunoprofiling, this part goes for spatial transcriptomics, now you can start interpolating between the two to ask: what are the molecular drivers for these distinct patterns of spatial infiltration? You get a chance to go from observation to mechanism, and hopefully that reveals something about disease biology that we wouldn’t have otherwise been able to appreciate.
Ross Katz: With that in mind, I want to jump to your second paper, which takes a different but complementary approach. This paper starts with pathology images like H&E slides labeled with the clinical outcome and then uses those to discover the morphological patterns associated with drug resistance, mapping those patterns onto spatial transcriptomics and ending up with drug targets. Can you walk through that entire logical progression — from a stained tissue slide to recommending a CDK inhibitor?
Arvind Rao: Before getting into it, I want to call out our brilliant graduate student Reva Kulkarni who led the work. What she wanted to ask was this: if you have H&E images from patients and their corresponding outcomes — either as a consequence of treatment or what happened to them eventually — that’s a very robust set of data for which you have both the imaging and the outcomes available simultaneously. H&E images are the lingua franca of pathology, getting them has been fairly standard, and the price point is very affordable. Doing it at scale is not hard. For some subset of those H&E images, one can then go and try to do spatial transcriptomics. And because the price point for spatial transcriptomics is at least two to three orders of magnitude more, it’s impossible to get the same volume of data having both the H&E and the spatial transcriptomics. I will use the abbreviation ST for spatial transcriptomics. Because you don’t have this tripartite combination of high ST data, high H&E data, and high outcome data, we need to borrow intelligently. What do we do? As a first pass, we use this combination of H&E data and outcome data to find those morphological patterns most associated with outcomes — in our case, treatment resistance. Then we ask: looking at the patterns identified in those H&E images and the companion ST data, what is the interplay between the ST-derived molecular mechanisms and the patterns found to be associated with those disease-level outcomes? This lets us go between these three vertices of the triangle, but leveraging what’s available. If you did it any other way, you would have a huge sample imbalance between the ST data and the outcome data — almost two to three orders of magnitude because the price point comes into play. When you do it like this, you can then nominate molecular mechanisms and hopefully actionable targets that connect to: why does treatment fail, what’s unique about this patient, what’s the molecular mechanism responsible for it, and what drugs might best modulate things to become more favorable? That was the approach we took in linking up these different modalities, which had vastly variable numbers of samples to go with them. That was our engineering trick to go between these three and still get something useful.
Ross Katz: If I’m recalling this paper correctly — correct me on anything I’m missing — there is a portion of the modeling exercise where you’re using a generative adversarial network to use the H&E slides to reconstruct what we believe the spatial transcriptomic profile would look like in that case. Am I remembering that correctly?
Arvind Rao: The way we thought about it was that GANs, generative adversarial networks, can be used in a variety of different ways. One is for augmentation of the data — for synthesizing examples that look like the real ones without necessarily being new ones. We used it more in the processing chain, not for any specific mechanistic intent but more as a way to build this tripartite workflow in a more harmonious manner.
Ross Katz: Another aspect of this paper that I found really interesting was the multiple instance learning approach, where you were segmenting the slide images and then utilizing the attention weights from the model you’re training to identify the patterns in the different instances associated with a clinical outcome of record. What that allows you to do is interrogate those portions of the slide — or in the case of validating your work, go to a domain expert in oncology and ask: what’s going on in this portion of the slide? That lets you verify that the model is in fact picking up on real biological mechanisms. Am I remembering that correctly, or how would you fill in the gaps?
Arvind Rao: Absolutely right. I also want to point out that we as engineers built on top of things that have become a lot more available. Some labs have been very nice about being responsible citizens for science and put out very good software that we can build on top of. The idea of the attention-based identification of interesting regions and the MIL was actually built on top of a tool called CLAM from the Mahmood Lab at Harvard. They’ve been fantastic at putting out solid reproducible code, so we built on top of that. I can’t take credit for the MIL genius in putting that into this framework and finding useful patterns, but we built on top of that to then go to regions implicated in treatment resistance and map that back to ST data to understand: why do we see what we see?
Ross Katz: After you answer the question of why you see what you see, my understanding is that by mapping it to the spatial transcriptomics — because you have what’s happening genetically — you’re able to nominate specific potential targets that might be implicated in the biological mechanism, and then interrogate what drugs exist today that might be able to inhibit those targets. Am I thinking about that right?
Arvind Rao: Absolutely. As engineers we build a lot of things on top of what’s already available. The idea of going from the prioritized molecular mechanisms to networks was done thanks to a variety of tools, both open and commercial, that we were able to leverage. People have already taken the trouble and done the hard work of organizing things into pathways, organizing things in terms of which drugs affect what pathway. We can look at these databases and put it all together to say: what’s the connection between this nominated pathway, the target, and the drug we want to go after? There’s been a lot of community help, and we want to be clear about that. We just hope that this excellent spirit of collaboration continues in the community, because then we can riff off of each other to build tools that we can eventually use to help patients. It’s a great example of good quality tools being made available publicly and others building on top of them to answer questions.
Ross Katz: Yeah, and also understanding what datasets are available, what tools are available, and how you stitch the tools together in a way that is methodologically rigorous — computationally efficient but also validatable from a biological perspective. Your work feels like it’s generally there.
Arvind Rao: I think you mentioned in passing something extremely relevant to what we are seeing in biology right now. There’s a lot of solid work tending to go towards multi-agentic workflows — things where you can spin up a whole end-to-end pipeline that threads together multiple agents that do different things. But if the discipline around ensuring that these things are cross-calibrated, that they talk about the same mechanisms and do things the way they ought to be done, is lacking, then it’s very easy to amplify the noise and not the signal. What you just said contains a very deep insight: when you start building these multi-agentic workflows in the context of all the amazing AI models today, it’s even more important to understand where these models were manufactured, where they went to school, what kind of architecture was used to train them, and to ensure that there’s an impedance match between these different workflows. Otherwise, garbage in, garbage out.
Ross Katz: That brings me to another question: when you see your students or other people in the community trying to merge these different data types together and bring them under a single predictive framework, maybe with agentic AI in the middle, what are the common mistakes that you see? Where do things tend to go wrong?
Arvind Rao: The key part is that when you start putting these things together, what we’ve learned is that we are very cognizant of the fact that we don’t have some of these very deep knowledge pools to understand where the noise might be springing up from. Talking to the data acquirers to ask where’s the noise, what could be the different batch effect parameters, etc. — that becomes really important. You have to pre-process them in ways that match the structure these models then ingest. That engineering is non-trivial. Then once you use the models and come up with a prediction, you need to think about calibrating that prediction with some concept of uncertainty — data-based, model-based, or even plausibility-based. You need the sniff test to ensure that what you’re getting is plausible. That’s where collaboration with a domain expert becomes really important. And once you’ve made the inference and hopefully have some measure of uncertainty, that uncertainty measure can then inform how best to do the next experiment, how best to design the next model, how best to acquire data that is most amenable to improving the model. From the perspective of building gradually better models, the scientific method becomes even more pertinent than what we’ve seen in the past with the use of AI.
Ross Katz: We’re in an environment where the AI tools available make it easier than ever to boot up whatever modeling software is out there and believe that you can prompt Claude or ChatGPT into curing cancer, and the nuance of how that gets approached is really important. Returning to your research — the two papers we just talked about feel like complementary pieces of a larger vision. There’s SPIFEE building richer representations of how cells and pathways interact spatially, and then you’ve got this multimodal framework that connects the spatial patterns to molecular mechanisms and drug targets. Am I reading that correctly? Where does the overall pipeline of your research and your students’ research go from here?
Arvind Rao: What we used to call average ability has now gotten amplified beyond our wildest expectations. What used to take months to code up a workflow has become a lot more accessible. You can build on the power of compute, stand on the shoulder of giants, look at all these GitHub repos and build up a new framework pretty quickly. We said: with all that’s happening, where can we have the best impact? Let’s now start building different pieces of this overall pipeline and see where it needs to go from biological understanding to biological insight to interpretation to the clinical and translational component. The two papers we discussed today get at this in two pieces — one builds better representations of the spatial biology, the other connects those representations to mechanisms and interventions. The long-term goal is a pipeline where you can go from tissue to mechanism to decision in a way that’s traceable and defensible. That’s where we are trying to put all these pieces together.
Ross Katz: That makes a lot of sense. Just to tease that out — my understanding is that because you’re reconstructing this much slower and more expensive data in spatial transcriptomics from the H&E slides that are widely available and very inexpensive, and you’re paying very close attention to the traceability and inferential rigor of how the decisions are being made, you’re setting up a stage where computational tools using existing data can enter the clinic and enable practitioners to be that much more efficient and effective at understanding what’s going on with a given patient — and at understanding the likely impact, assumptions, and caveats that need to be kept in mind when potentially prescribing an existing or new drug to a particular patient. Am I reading between the lines too much?
Arvind Rao: Beautifully put. Everything you’ve said is completely in line with exactly what we are talking about here. Yes.
Ross Katz: To switch gears a little bit — since you’re advising clinicians and hospital administrators on AI, you’ve described yourself as a cautious optimist about AI in this space. Can you describe what specifically makes you cautious and where you see the most excessive exuberance relative to what the evidence supports in how AI is applied in the computational biology space?
Arvind Rao: My biggest concern is that perhaps we are very good at demonstrating capability but not reliability. We are very good at saying this can be done and then showing it. That’s the paper, and that’s great. But the translational gap between what that paper shows and a reliable product that can be used every day in the lab or in the clinic — I don’t see that happening as much. What I’m seeing is we are building impressive models, each one with a billion more parameters, each one with a more expansive dataset, but the concept of going from those predictions to consistently trusting them in high-stakes settings like healthcare — that part we could do a little bit more work on. What I mean is the metrology of AI. You take a simple ruler, a slide rule, a calipers — you make a measurement and you can say your height is 5 feet 7 plus or minus so many inches. You’re very good at that because those things matter. When I make predictions and I don’t have an error bar, there’s no metrology to go with it — what do I do with that prediction? If your prediction is this person is COVID positive with a 3% error rate, that’s a very different thing than saying this person’s COVID positive with a 50% error rate. COVID positive status is helpful, but accompanying it with some calibration of what that prediction is associated with becomes really important. It also feeds into reliability. The stochasticity normally associated with AI can give results that are all over the place. Run your favorite generative AI engine twice, and unless you are very deliberate about the prompts to ensure consistency, you can get vastly differing results — not because that’s magic, because that’s math. Generative AI by its definition is generative, it samples from a distribution. The variability you see is very natural. But you can’t have that kind of variability when it comes to treating patients. When you start building workflows that have multiple generative AI agents all threaded to each other, it becomes really difficult to talk about credentialing, robustness, and uncertainty and error propagation. Capability demonstration is very nice, but we need to be more deliberate about how we test these things in real-world settings responsibly to ensure they get to a level of quality where you can deploy them.
Ross Katz: Getting to the idea of evaluating — there are a lot of foundation models for biology coming out and they’re getting a lot of attention and investment. How do you think about evaluating whether they’re advancing clinical utility or just gaming the benchmarks?
Arvind Rao: This reminds me of Goodhart’s Law: when a metric of performance becomes your cost function, it ceases to be a metric. A metric evaluates performance after the fact. If you use the mean squared error or some version of that in your training process itself, then it’s not even the right metric to evaluate your algorithm by. I think the field has fallen into that trap. You’re chasing the accuracy metric because that’s what everybody likes to see. But what information are you actually extracting? Are you extracting signal or artifact? How does performance degrade under real-world shifts? Simple things — a different scanner, a different protocol for building an H&E slide, some stray marks, pen marks, tears, or glass breaks — can massively impact the quality of your prediction if you didn’t adjust for those confounders going in. When you build an algorithm, is it built on a nice pristine dataset, or are you actually using the real-world confounding information to build a more robust model? That’s the second thing I like to ask. The third: does it actually change a clinical decision in some tangible way versus just being a benchmarking exercise? I scored five points more than you on AUC — super interesting, but did it tangibly change something for the patient or how you would deliver care? That brings you to real-world questions during deployment. People have already shown this: you had this sepsis algorithm that could detect the onset of sepsis quite early, but it began giving so many alerts so often that it led to alert fatigue and the people in charge just shut it down. Going from building an algorithm that meets some benchmark to actual deployment for value — those are different tasks. If you can’t answer these questions, you are likely falling into a trap of hype and not real utility. When you have all these great foundation models, I have to ask: what is the difference in the information you’re recovering that truly justifies this additional billion parameters? If the change in that vector is not that much, what purpose is it truly serving?
Ross Katz: That maps to a lot of my experience with machine learning and AI projects. Fundamentally, if you don’t know what decision this is going to change and how it changes the quality of that decision, the quality of the model is impossible to measure. It’s great that you seemingly know a lot more information about what’s going on in your environment, but if that doesn’t change any of the outcomes — even if the information is factually accurate — it is irrelevant. It’s noise from the perspective of the goal you’re trying to accomplish. So you’ve been building AI literacy programs for healthcare professionals and pharma teams. Is there a core gap you’re trying to close, and what does that gap cost when it goes unaddressed?
Arvind Rao: I’ve been on both sides of the table — both in terms of model articulation and procurement. What I was observing was a gap in the articulation of what it takes to go from a model that meets all the benchmarks and metrics to deploying them. The ability to evaluate AI and appreciate its failure modes is probably as important, if not more important, than knowing how to build an AI engine. Thanks to tools like Codex and Claude-Code, you can go from an idea to a vibe-coded prototype pretty easily now. But if you don’t appreciate how to evaluate it beyond the standard metrics, or to evaluate its failure modes, a gap comes in. That gap can either purposefully or inadvertently lead to organizations buying the wrong tools, misinterpreting the quality of the output, misinterpreting vendor claims about the quality of the data and the model. When that misinterpretation happens and there is a system failure, it causes a loss of trust. The problem was not in any particular component — it’s in the entire workflow. It’s not a technical failure as much as a decision-making failure. The literacy gap can lead to bad procurement decisions, ungovernable systems, and misplaced trust in outputs. My goal with this literacy exercise is to educate the community about potential failure modes so they can make better choices, procure knowledgeably, and deploy responsibly to help their patients. AI education for building models is all great and there’s plenty of it already, but I see a genuine gap in how you build the right rubric for a sniff test that anybody — even non-AI experts — can use to meaningfully calibrate whether what they are being sold is not snake oil.
Ross Katz: That’s really interesting. Are there any nuggets from that work — things that you might be able to pick up with your sniff test that healthcare and pharma teams aren’t?
Arvind Rao: My most common question is: what is the uncertainty accompanying your prediction? If they can’t answer that in a way that’s satisfactory — it doesn’t need to be very fancy — but if you don’t even have an answer to how you do it and what’s the basis of that uncertainty quantification, I know immediately that the numbers are not the entire story, there’s more under the hood. The other question I like to ask is: that’s all very nice about where it does well, but where does it fail? And are there patterns to those failures that can help us understand what this model is capable of doing or not? Is it attributable to data shift artifacts like instruments, scanners, and sites? Is it something more serious about the wrong population for which it is being recommended? And what does it mean in terms of real-world evidence for the deployment of what you’re predicting about and its translation to actually helping a patient? You don’t need to be an AI engineer to ask these basic questions. When I go to buy a car, I’m not an automobile engineer. Yet there are basic questions I know to ask: what’s its mileage? What tires does it need? Does it drive well in the snow? I don’t need to be an automobile engineer. Similarly, when I go to buy my favorite cereal at the grocery store, I’m not a farmer and I don’t know the first thing about food processing. Yet there’s enough information on that box that when I turn it over I can say: here’s the ingredients, here’s where it was made, here’s the guidelines it adhered to while producing it. I can say: it’s safe enough for me to consume regardless of its price point. That’s something I eat, this is something I drive, both are related to my safety. When I use AI to do similar things, how come nobody asks those questions? It should be explainable to anybody using it. Tomorrow your patient is going to come up to you and say: Doctor, why did you use this tool on me? And why would you not use something else? If you can’t give a good answer, we are in trouble. The vendor has to be held accountable to those important questions, and any procurement exercise has to be informed from that perspective — not just relying on nice ROC curves or accuracy metrics. That’s a good start to a conversation, but it should not be the end.
Ross Katz: The accuracy is sort of the question of whether we should be having a conversation at all — it’s not the end of the conversation, it’s the start. And as I’m hearing you describe things, as individuals when we pick up a tool like a pen, we recognize that we have to learn to hold it to get reliable letters or numbers out of it. As organizations, there’s a variety of questions that need to be asked — particularly about failure modes — to teach the organization how they need to hold that model or computational tool in order to compensate and ensure that even if the model is 95% reliable, you’re able to get near 100% of what you need out of it by compensating for the 5% of cases where the model might fail. As we head toward the end here — you work in radiation oncology, which has a long history of quantitative models and treatment planning. How does the culture of that discipline, where physics-based models and dosimetry are already deeply embedded, shape how AI tools get received there versus other areas of oncology where quantitative approaches are less native to the workflow?
Arvind Rao: I’m very lucky to be in a radiation oncology department because their knowledge of physics and mathematics is genuinely very well aligned with the care that they offer patients. Things like segmentation errors, dose planning, the physics, the calibration — every morning or every so often they will calibrate their devices. All of this speaks to a discipline. The key part to remember is that to deploy these tools to the extent that you want to see success, there needs to be a discipline to the calibration. When you go to a biology lab, they calibrate their pipettes every so often — somebody comes around and says we’re calibrating pipettes, come do this. Where does that calibration happen for AI? We talk about drift. It is a natural thing because we leave these things unattended for too long. Radiation oncology and radiology have adopted certain technologies much earlier, and their hard-learned lessons can be very big beacons for how we need to do things in a more principled way. What will it take to graduate a device from a curiosity to a deployable entity? Because these communities are so well in tune with what it takes to go from a proof of concept to a deployment that can help patients, they can become really important fact-checkers and guides for how to do things right. I’ve learned a lot from my radiation oncology colleagues.
Ross Katz: That’s great. Well, I think we’ll leave it there. Arvind, thank you so much for coming on the podcast today. I really appreciate the time and look forward to connecting down the line.
Arvind Rao: Absolutely. Thank you for having me and for all the great questions. Thank you.
Ross Katz: Arvind, thank you. This was a masterclass in what it takes to move from multimodal data to clinical insight and why the gap between building AI tools and deploying them responsibly deserves far more attention than it gets. The core takeaway here is that the hardest problems in computational oncology aren’t algorithmic, they’re about data quality, validation rigor, and making sure that the people who deploy these systems understand them well enough to govern them and use them responsibly. You can find Arvind on LinkedIn and explore his lab’s work at the University of Michigan, especially if you’re thinking about AI literacy programs for your own team. If you know a data leader or a clinical team that’s evaluating AI tools for biotech and needs a sharper lens for what’s real and what’s hype, send them this episode. And if you haven’t already, please subscribe so you don’t miss what’s coming up next. We’ll talk to you next time.
Jason: And that’s it for this episode of Data in Biotech. If you enjoyed the episode, please subscribe, rate, or leave a review in your podcast platform of choice. See you next time.
