Listen on
Overview
Most production AI evaluations collapse model performance into one or two aggregate metrics across the entire test set. In low-stakes settings, that is acceptable. In clinical AI, where a false negative means a missed lesion and a false positive means an unnecessary procedure, aggregate numbers hide the failures that matter most. An F1 of 80 percent across every dental X-ray a model sees can mask the fact that the model is wrong on every image from the third-largest sensor manufacturer, or systematically blind to early-stage decay on anterior teeth. The same flawed sampling that creates the blind spot in training data usually creates the same blind spot in the test set, so the test set ends up ratifying a model that is failing in production.
Sadegh Salehi is Director of Research and Principal Scientist at Overjet, where his team builds FDA-cleared vision models that detect and quantify dental disease across billions of X-rays from thousands of practices. The data problem has a staggering number of dimensions: 32 anatomically distinct teeth per adult patient, multiple image types (bitewing, periapical, panoramic, CBCT) capturing different anatomy, 15 to 20 sensor manufacturers producing perceptually distinct images, and disease severity ranging from barely visible early-stage caries to obvious pathology. Sadegh’s academic work on loss functions for imbalanced medical image segmentation has been cited more than 3,000 times. At Overjet, he has applied that expertise to evaluation, architecture, and the regulatory loop that surrounds both.
In this episode, host Ross Katz and Sadegh walk through how Overjet evaluates models across subgroups instead of by aggregate metric, how the architecture evolved from more than 20 narrow task-specific models into a single foundation model the team calls Unity with independent prediction heads on top, how Overjet became one of the first companies to secure the FDA’s Predetermined Change Control Plan, and how dental treatment plan procedure codes give the team a noisy but real production monitoring signal that most ML teams do not have.
Key Takeaways
Aggregate metrics hide the failures that matter most in high-stakes AI
An F1 of 80 percent is a meaningless number if you do not know what it is 80 percent across. Sadegh’s team trained a decay detection model mostly on molars, which are thick teeth that show decay as a subtle gray shade on the X-ray. Run on anterior teeth, which are thinner and naturally show more gray, the model flagged everything. The aggregate metric did not surface the failure mode; it buried it. The fix is structural: evaluate by sensor manufacturer, image type, tooth number, and disease severity, not by a single combined number. Then push that subgroup awareness back into training with an adaptive loss function that watches subgroup performance and adjusts during training, so the model is pulled toward weaker subgroups without overfitting on the larger ones.
The same flawed sampling that creates gaps in your training set creates them in your test set
A model learns the most efficient signal in the input that predicts the label. If every image from a rare sensor manufacturer in your training data happens to come from a healthy patient, the model learns that the sensor identity means healthy and skips the underlying visual task. In production the shortcut shows up as systematic false negatives on that sensor’s images. Aggregate metrics will not catch it because your test set was sampled the same way as your training set, which means the test set has the same blind spots. The shortcut shows up only when you break evaluation out by the dimensions that matter when the model hits production: sensor, image type, anatomy, disease severity.
Overjet’s foundation model architecture is also a regulatory architecture
Overjet started with more than 20 narrow models. One for fillings on bitewings, one for crowns on panoramic images, one for implants on a specific sensor. Each was trained on thousands of images and each was safe, but the coverage was limited. As the dataset grew to millions of images, those small models generated noisy labels that became the training signal for a much larger backbone, which Overjet internally calls Unity. The team verified Unity had learned to read X-rays rather than statistical patterns by using its latent space to retrieve the same pathology across different sensors, image types, and patient populations: extract the latent vector for a molar with significant decay from a bitewing, run it on PAs, and the retrieved teeth show the same abnormality across the heterogeneous sensor set. They then built independent prediction heads on top of a frozen Unity backbone, one per clinical indication. Independent heads were as much a regulatory choice as a modeling choice. With the FDA, every cleared model must stay locked, so an architecture where retraining one head would require reclearing every other head would have been a dead end.
Learn the artifact, not the image, when ground truth is impossible to label
Different dental sensors have different post-processing software that produces different image quality, and Overjet sees the raw image, not the post-processed one. The team needs consistent image quality across all sensors so downstream prediction is reliable, but you cannot annotate “good image quality.” No one is going to paint a clean image by hand. So Overjet trained an unsupervised residual model that learns the distribution of artifacts (noise, contrast distortion, sensor-specific interference) across the heterogeneous sensor set. The model’s output is the artifact, not the image. Subtract that artifact from the raw input and you get an enhanced image. The inversion makes the supervision problem tractable. You do not need labeled examples of good images. You need a model that can recognize what to remove.
The FDA’s Predetermined Change Control Plan changes the cadence of medical AI iteration
Before PCCP, every meaningful update to a cleared model meant restarting the regulatory process. For deep learning teams capable of training a better model in three months, that was a structural barrier to improvement. PCCP, introduced by the FDA around 2020 to 2021, lets companies define up front the scope of acceptable changes (evaluation criteria, subgroup thresholds, intended-use boundaries) and then update the model without filing a new clearance, provided every predefined condition is met. Overjet was one of the very first companies to clear a model with a PCCP. The result for indications covered by PCCP: retrain on more data, run the comprehensive evaluation, pass the predefined thresholds, push the new weights to production. The audit trail still has to be complete; the gate just moved.
Treatment plan procedure codes are a noisy but real production monitoring signal
After a dental appointment, the dentist enters CDT procedure codes for the work the patient agreed to: a filling on the mesial side of tooth number two, a crown on tooth fifteen. Overjet has access to those codes, which gives the team a feedback signal most production ML teams do not have. If the model detected decay on a tooth and a filling code appeared on that tooth after the visit, that is a noisy true positive. If a filling code appeared on a tooth the model did not flag, that is a candidate false negative, with caveats. The decay might have been on an occlusal surface that X-rays cannot see, or the dentist might have caught it on direct examination. The signal is noisy enough that you cannot run an evaluation off it alone, but concrete enough to sample from when you want a real-world check on whether the model is drifting.
Related: CorrDyn works with companies building machine learning systems against heterogeneous, high-stakes data, and partners with biotech, healthcare, and life sciences organizations on AI strategy, data quality, and the evaluation and monitoring decisions that determine whether a model holds up in production.
Full Transcript
Jason: Welcome to Data in Biotech, a podcast from CorrDyn where we explore how companies leverage data to drive innovation in life sciences. Every two weeks we sit down with an expert from the world of biotechnology to understand how they’re using data science to solve technical challenges, streamline operations, and further innovation in their business. Here we go.
Ross Katz: Welcome back to Data in Biotech. I’m Ross Katz. Here’s something that happens more often than anyone in AI wants to admit. You build a model, you evaluate it, the numbers look great: high precision, strong recall, everything checks out on your test set. You deploy it, and within a week someone’s asking why it completely falls apart on a subset of data that you never thought to isolate. In most industries, that’s an expensive mistake. But in healthcare, it’s a different kind of problem entirely. When your model is analyzing medical imagery, making detections that influence whether a patient gets treated or sent home, a blind spot in your evaluation isn’t just a performance issue; it’s a clinical risk. And the challenge gets harder when you consider what medical imaging data actually looks like in the real world. It arrives in dozens of different sensors and acquisition systems, each producing images with different contrast, resolution, and noise characteristics. So, for example, in dentistry, a bitewing X-ray, which is the close-up shot your dentist takes of your back teeth, looks nothing like a panoramic image that you get of your full jaw. A digital capture from one sensor manufacturer looks nothing like one from another. And yet the model has to work across all of them—reliably, fairly, and under the watchful eye of the FDA. My guest today lives in the middle of that problem. Sadegh Salehi is the Director of Research and Principal Scientist at Overjet, where he leads the team building FDA-cleared vision models that detect and quantify dental diseases across billions of X-ray images from thousands of practices. His academic work on loss functions for imbalanced medical image segmentation has been cited over 3,000 times. And at Overjet, he’s applied that expertise to a data problem with a staggering number of dimensions. So, you’ve got 32 teeth per adult patient, each with different morphology, multiple image types showing different anatomy, and then you’ve got 15 to 20 sensor manufacturers in the field, with 90 percent of practices using just three of them. And disease severity distributions that range from barely visible, early-stage decay to obvious pathology that anyone could spot. In this conversation, Sadegh walks us through what it actually takes to evaluate models across all of those dimensions, and why aggregate metrics can mask catastrophic failures on specific subgroups of your data. He explains how his team evolved from over 20 narrow, task-specific models into a single foundational architecture they internally call Unity. How they use treatment plan procedure codes as a noisy but real feedback signal to monitor model performance in production. And how Overjet became one of the first companies to secure the FDA’s predetermined change control plan—a framework that lets them update models without filing a new clearance every time. So whether you work in medical imaging or not, the evaluation traps that Sadegh describes apply anywhere you’re building models against heterogeneous, high-stakes data. Let’s get into it. Sadegh, welcome to Data in Biotech. Can you give us an overview of what Overjet does and what your role looks like day-to-day as Director of Research?
Sadegh Salehi: Thank you, Ross, for having me in this podcast. So Overjet is a healthcare AI company focused on improving oral health by bringing more objectivity and consistency into dentistry. We build learning-based models that analyze different dental modalities—it can be X-rays, it can be CBCT, 3D X-ray images, or even voice, it can have different modalities of the intraoral images, intraoral scan images—and help clinicians detect condition like cavity, bone loss, other pathologies earlier and more accurately. And my role as the Director of Research here at Overjet is to make sure that we can train, develop, and evaluate these models rigorously and make this happen in the real world.
Ross Katz: Yeah, that’s awesome. And just so that I’m understanding Overjet’s model a little bit, my understanding of the value proposition is that from the clinician perspective, you’re able to detect conditions more accurately by using all of the different modalities of data that you’re collecting, and then from the patient’s perspective, my understanding is that oftentimes you get to see the indication of what your condition is, which gives you more trust in the condition that’s being assigned to you, and then also from the payer perspective, there’s a closed-loop documentation there. Am I thinking about that right, or how does that work?
Sadegh Salehi: Exactly. So you as a dentist, you see something in the X-ray image, you see something when you are examining the patient, and now you want to communicate with the patient about the treatment plan and what procedure you are going to apply on that specific tooth because of these indications that I can see as a trained dentist in the X-ray image. And it is easier for the dentist to communicate with the patient to show that, “Okay, you see that small red thing that Overjet AI detect? That is decay, and if I turn it off, now you see that gray shade that you see on that tooth? That is decay. And if Overjet didn’t color-code that part, it is really hard for the untrained eye to see that. The dentist saw it, but the patient didn’t, and it is really hard for communication from the sense of the dentist and patient that, ‘Okay, that part is very small, sure, but it is going to be bigger, so it is going to be a problem for you, so now we need to do the treatment.’ But having those quantification, having those color codes on top of the image, it just makes trust for the patient to realize that, ‘Yeah, okay, that is serious, and I need to do this procedure.’”
Ross Katz: Yeah, awesome. So you’ve got billions of dental X-ray images and data coming in across all of these modalities, and what you’re doing in the middle is using machine learning to detect these very hard-to-see, especially for the untrained eye, conditions that are inside of people’s teeth and oral cavity. Can you give us an overview of what the data looks like and the different variety of sensors, imaging modalities, and acquisition systems that you’re dealing with as you’re building those?
Sadegh Salehi: Sure. So even in one modality, there are so many variations. If you want to think about the different modalities in dentistry that we’re actively analyzing, we have one main modality that is X-ray, dental X-ray images. Those are the X-ray images from the tooth. And even dental X-ray images have different variations. We have bitewing images, we have PA images, periapical images. When you take the bitewing image, you don’t see the root of the tooth, but when you take the PA image, you only focus on one part of the jaw, not two jaws at the same time, not maxilla and mandible at the same time. Or when you take the panoramic image, you are looking at the whole jaw at the same time, but now you have lower resolution of each tooth. So you have higher resolution for each tooth when you take the X-ray bitewing and PA image, but lower resolution but more context when you take pano. These are all different variations of the same modality, just X-ray. These are 2D images. We have CBCT; it is a 3D image. The jaw and teeth in the jaw are all 3D objects, and these images are just representation of that 3D object. In CBCT we take 3D, so now you have depth in that image. But then you have lower resolution, you have higher exposure, so not always the clinicians take the CBCT. So that’s a different modality.
Ross Katz: Okay, so when you’re training a model to detect something like tooth decay across all of these different modalities and distributions, what are some of the hidden subgroups in the data that you need to think about that might undermine the performance of a model if you’re not paying attention to them and accounting for them when you’re building the model?
Sadegh Salehi: That’s a great question, specifically in the dental X-ray domain. One main difference is when you take the chest X-ray, you have one image, one 2D representation of a 3D object, and that 3D object is the chest. But that’s one representation, one image, and your model is analyzing that one image. But in dental X-ray, you don’t have just one representation. When we take these images, you have at least 32 objects. For adult cases they have 32 teeth, or when you have a mixed dentition, it can go to 40 or more because now you have the pediatric tooth, and underneath the pediatric tooth you have the erupting adult tooth. These are very weird-looking images. But these are different teeth, and these different teeth have different representation in the X-ray images. Think about your molar teeth—they are very thick teeth—which means that when you take your X-ray image, those photons are just not coming through that tooth, which means they look more white, they look more radiopaque in your image. But you have the anterior teeth that are thinner compared to the molar ones, which means that you see more shade. And what was that shade? That’s a decay. So most of the time you say that when you see a shade of gray, that indicates decay, something is wrong, let’s train the model on that. But if you train your model only on the shades of the molar teeth, now every single shade on your anterior teeth is detected as decay. Why? Because it is a thinner tooth. So if you don’t introduce this variability between tooth numbers in your evaluation, you’re just evaluating the wrong thing. To give you an example: assume I have my model and I’m training a decay model and I’m getting the F1 of 80 percent. That’s a very high number, that’s a great number. But all your training and test data are coming from molar teeth. Is that a good model for anterior teeth? Probably not. And we did these experiments just out of curiosity—“Okay, what would happen if we do that?”—and you will see that you have a very bad model that has so many false positives on anterior teeth because we only trained on the different distribution of data. So these are different dimensions where, when you want to train your model, you want to make sure that these subgroup dimensions have representation. You have the sensor type, the image type, the acquisition type—these are metadata subgroups. But inside of the image itself, you have different distribution of the tooth numbers that you want to make sure you have representation of in your training set. And not only that, you want to make sure that the prevalence of your data is covering those distributions, which means you want to have some anterior teeth in your training data that has decay. So you want to show these distributions to the model when you are training it.
Ross Katz: That makes a lot of sense, and it makes me wonder whether there’s a preprocessing step where you’re segmenting the different images and modalities into the teeth and assigning them numbers so that you know which tooth you’re dealing with, or is that kind of abstracted away in the modeling process—basically you want to make sure that you know which teeth are in the different images so that you can sample effectively as you’re going through training and evaluation, but you let the model learn how to detect tooth decay across all of the teeth in the image without having to handhold it through the segmentation process?
Sadegh Salehi: During the segmentation of the decay, for example, we do not provide the information of the tooth number to the model. We just let the model learn about the different representations of, for example, decay in different teeth. But during the sampling of the data for training, we are very well aware of the representation of these different tooth numbers in our training and test sets. We are very well aware of the image type distribution, we are very well aware of how many—we have millions of images—20 percent of them contain these anterior teeth, 20 percent of them these—so you need to have this information because when you do the evaluation afterward, you want to make sure that it is following reality, following the real distribution out there. You make sure that you train in the same distribution as the real world. On the sampling side, we are very well aware of these things. But during the training, we do not want to bias the model about this information. We just let the model learn.
Ross Katz: Right, and my understanding is that these images that you’re getting are already—it’s a supervised learning problem. You already know the clinical diagnosis, or lack of clinical diagnosis, that’s associated with each image. So without necessarily needing to tell it “This is where tooth number 14 is located in this image,” it’s able to learn from the quantity of images that tooth number 14 has decay happening roughly here. Okay. Exactly. I’m interested in how you think about sampling during the training and testing process in order to make sure that the model is getting that representative distribution but also that you’re maximizing the combination of precision and recall that you want to get out of the model.
Sadegh Salehi: For every single subgroup of your training, you have subgroups of your training data and subgroups of your target. What am I going to detect? Am I going to detect the large decays, or am I going to detect the calculus in this image? These are different labels. So you want to make sure that you have a good and representative prevalence of each of those targets on each of your subgroups, in a way that you can then claim, “This model can detect decay on this subgroup, and I already tested and evaluated that. Now I can provide you the number.” So now I can, with peace of mind, sleep at night and make sure that my model is doing what I am claiming. That’s a very important part that even the regulatory space is asking us to provide this information. And that’s the whole point. To give you an example—let’s focus on some metadata subgroups, different sensors. If you sample 1,000 clinics and check which sensor they are using for taking X-rays, that distribution is a heavy-tail distribution. Most of the clinics are using a couple, three, four different sensors—90 percent of them are using these sensors. But then you have 10 percent that are using very rare sensors out there, which are visually different—it is literally a different image when you look at it. If you don’t sample them correctly—assume you just sample randomly from your clinics—then 90 percent of your data is on the well-represented sensors, the ones that are frequently being used in the clinic. But then, assume that if you are not aware of the prevalence of your rare sensor, and all of your rare sensor images are healthy. There is no decay in that image. So what does your model learn? Your model learns that for this sensor, there is no way you can see the decay, because it has never seen any decay in that subgroup for that rare case. I’m pretty sure any scientist that trains a model knows that these models are looking for shortcuts. They are not doing what you want them to do. They are not looking at what you want them to see. You need to guide them to see what you want them to see—and they are really good at finding shortcuts. If all of my rare sensor images are healthy, my model just doesn’t look at the image at all. It just says, “Okay, this is coming from this sensor,” because it’s easier to classify by sensor rather than finding that shade in the image. “It’s always healthy.” And then you put this model into production, and a clinician comes back and says, “What’s wrong? You have so many false negatives. You always say it’s healthy.” And then you go look at your training data and say, “Oh, now I see that this subgroup had a prevalence of zero for this abnormality.” Now you understand, “Okay, I need to make sure that this has the correct prevalence.” So that’s the hand-wavy explanation of why it’s very important to be aware of the prevalence on your subgroups.
Ross Katz: Right, you have to be aware of the prevalence on the subgroups. I’m also interested in how that plays into the training and evaluation process. When you talk about the subgroups of your input dataset and the subgroups within your targets, you’re talking about a massive degree of cardinality that is going to be really hard to pick up on, and so I’m imagining that you have to prioritize which intersections of subgroups you care most about and also think about how you put that into the loss function, how you put it into your evaluation—how you teach the model to also care about all of the subgroups in that stratified manner that you’re talking about.
Sadegh Salehi: That’s true. One thing that’s really important specifically on the evaluation side is that you should not decrease the dimension of your metric to only one or two numbers. Just collapsing to one number—“Okay, my precision on these images is 80 percent.” What does that even mean? You are combining everything, all these different subgroups, into one number. You have 80 percent precision on what? You have 80 percent precision on the aggregated of all the subgroups of the metadata—different sensors, all different abnormalities, all different teeth. Is that enough to make a decision that this is a good model? I don’t think so. I have this discussion with my team several times a year: when you provide me only one number, I don’t like that. You’re zooming out so much that you don’t see any detail inside of your model, and you don’t know the performance of this model. So you need to give me the precision of each subgroup, the precision of each label. And not only that—when you are training your model, you want to have that loss function adaptive to look at these subgroups. First, you need to have a good representative dataset in your training. To be honest, that’s the most important thing. You need to sample your data in a correct way, in a way that you understand the distribution and the prevalence. But even then, during training, you want to have some adaptive loss function that looks at subgroup performance and adjusts the hyperparameter during training: “On this subgroup, we are not doing great, let’s push the model to look at that subgroup as well.” But in the meantime, we don’t want to lose the other subgroups that have more representation—you don’t want to overfit on the smaller one. This adaptive training is the key to make sure you are aware of these different representations and push the model to work well on all these different subgroups.
Ross Katz: Yeah, that makes sense. So when the model output comes out, you’re looking at the performance across each subgroup. But inside of the modeling process as well, you’re encouraging the model to attend to the performance across each of the subgroups—essentially updating based on how each subgroup is performing, understanding the benefit of one improvement in precision on one subgroup versus one improvement in recall on another subgroup.
Sadegh Salehi: Exactly. And not only that—sometimes based on the application, you want to have better recall than precision. “Okay, if I have a triage application, I want to have better recall. I don’t mind if I have some false positives, I don’t want to call every single one as positive, but I don’t mind having more false positives to avoid some false negatives because I care about the recall.” When you look at the application, you know which metric is more important for you.
Ross Katz: Yeah, so in the case of Overjet, you have expert clinicians reviewing the predictions that you’re making, and so obviously you don’t want to have a ton of false positives, but if there’s the occasional false positive, you expect the clinician to be able to differentiate between that and what’s actually there.
Sadegh Salehi: Exactly. They can differentiate; they will read—they always review our prediction. But also, if we can detect something that they just didn’t see, it’s good that, “Okay, take a look at this tooth as well.” It might be nothing, but take a look. And they appreciate it. It’s not an incidental finding, but pointing to other teeth in the image is good because now they can double-check.
Ross Katz: Yeah, that makes sense. And also from the dental practitioner’s perspective, they can go into the patient’s mouth and say, “These are the places where I need to be looking, and I can be reasonably confident that these are all of the places that are most important for me to be looking while I’m in here.”
Sadegh Salehi: Exactly. Exactly.
Ross Katz: Before we move on, you mentioned that the most important thing in order to sample effectively is to have the most comprehensive dataset you possibly can, and you’re—as I understand it—working with one of the largest datasets of dental imagery that is out there. But Overjet can’t have started with that dataset in place, so you sort of had to climb that learning curve in order to get to a point where you could solve problems with sampling that you might have had to solve with other methods earlier on. I’m interested in understanding that journey. How did you go from working with a smaller dataset and developing a clinically applicable model that’s capable of being FDA-approved to where you are now, where you’ve built up that cache of billions of images and you can start thinking about foundation models?
Sadegh Salehi: At the very beginning, we had different small models, each very specific for a different small subgroup. We had a model at the very beginning that was only working for decay on bitewing images for one specific sensor. That’s it. We could not cover everything. We didn’t have data, we didn’t have the annotation power—because when you have a primary model, then you can do model-assisted labeling. You provide the label, now annotators are correcting the label, so annotation is fast. But at the very beginning, you don’t have this. So you make sure that your model is trained for a specific image type, specific indication, and specific sensor. These are very small models. And then we had so many of these. We had more than 20 models doing these small things. One was expert for filling on bitewing. One was expert for filling on pano. All these different models, tiny models trained on not millions of images but thousands of images. They are doing decent, they are safe, they’re working as expected. But they are very focused on one specific indication and one specific image type. Then, moving forward two or three years, now you have a bunch of data. Millions of images. And these millions of images already have noisy labels, because the small models are doing their annotation. You can see that there’s an implant in this image because the implant model was there for that image type. Now you have millions of images with noisy labels. What you can do now is semi-supervise training of a large backbone that can take a look at every single one of these things at the same time. So we are moving toward, “Okay, now there is a foundation model here.” We moved to that foundation model more than three years ago. In our internal discussion, we call it Unity. Because we literally unite all these models into one foundation model. The way we thought about it was: all these X-ray images—pano, bitewing, PA, different sensors—they are all X-ray images. If you show an image from sensor one and sensor two to the dentist, they can read both. It’s not like, “I can read this image but I cannot read that one.” They can read. So what we were thinking was, “We need to have a foundation model that understands X-ray. We don’t want this foundation model to be expert on any indication. We just want to make sure that it understands the X-ray image and has the best latent space that we can use for different indications.” That was the main goal for Unity. We trained that Unity model—it’s a huge one—on millions of images with these noisy labels. And we figured out, “Okay, now this model is understanding the X-ray.” The way we realized it was understanding the X-ray without necessarily understanding the label was really interesting. We said, “Let’s use the latent variable of this foundation model and use that vector space to retrieve the tooth that has the same issue. For example, let’s find the latent variable for the molar tooth with huge decay. And now use that to retrieve the same teeth from other sensors, other patient populations, and even other image types.” We extracted that latent space from the bitewing, and then we ran it on the PA. And it is extracting the teeth that have the same abnormality, the same representation, just by looking at this vector space. We realized, “Okay, this foundation model is doing this job. It is understanding the X-ray; it knows what is going on in the X-ray.” And now we have different heads—the input for each head is that latent variable, and the output is a supervised prediction. Now you don’t need so much data because the model already normalized that for you. But those heads need to be trained in a supervised manner, with perfect labels. We do that, and now we have the foundation model being run on every single image that comes into our pipeline, and we have different heads—one head is expert in filling, one head is expert in crown, one head is expert in implant.
Ross Katz: Couldn’t we do that training together? Couldn’t we train all these heads at once? Yes, we can. But why didn’t we do that? Because if you have independent heads for different indications, you can change one head without affecting the other heads. We have a frozen backbone—that foundation model is frozen. But these heads, we intentionally make them independent from each other, so if we change something in one head, we don’t need to re-evaluate all the other heads. Yeah, that makes a lot of sense, and just hearing you talk about it, the journey is very fascinating. I just want to recount the insights that I got from it. When you’re in an organization that is smaller, closer to your startup environment, you’re operating in a resource-constrained and data-constrained environment, and you’re focused on the 80/20 of the problem. You’re trying to solve the problem as well as you can for the dental practices you’re actually working with using the devices they’re actually using, and you’re targeting only the sensor types being used by 90 percent of the dental industry rather than focusing on the long tail of imagery out there. You don’t have examples of all of the different disease types that you might want to detect, so you train the models you can train using the data you do have, and you’re building out the feature set of Overjet for the dentists you’re serving as you’re collecting more data and growing the organization. Then at a certain point, you’ve got these 20 models and sufficient data, and you can start thinking about a foundation model, using those 20 models to semi-supervise the foundation model to create that backbone, that latent space. So you’ve got this foundation model that embeds a single image, and the foundation is learning how to read these images across the full variety of dental issues that might arise. You want that backbone that is an expert at reading these images, and then you decide to have multiple prediction heads that sit on top of it, trained independently off of that backbone, so that your input space basically looks the same for every model but the way that model is interpreting it is different for each prediction head. And the reason you do that rather than having a model that predicts all of them is because there might be trade-offs in the quality of the prediction heads if you tried to train them all at once—they wouldn’t all get better, some would get better and some would get worse, and what you want is the best prediction head for each one. Am I thinking about that right?
Sadegh Salehi: Exactly. And also, we are in the regulated space. When we have the model evaluated and we say, “Okay, this is the performance of this model on these subgroups,” that’s it. You need to lock that model. We have this PCCP that we can talk about later. But in the regulated space, from the FDA’s point of view, if you have trained your model and you get the clearance for this model, you need to keep that model locked. You cannot change even one parameter of that model. So assume that I have three different clearances for three different indications. And now I realize that one of those indications has some bias, and I want to retrain that model. If I combine all of them in one head, when I retrain that, I now need to make sure that the performance of the other two indications are not affected, and I also need to get another clearance for all three, not just the one that has the bias. So that intentional separation between the indications was a very smart thing that we did—now that I look at it in retrospect, it just didn’t tie our hands by combining all of them together.
Ross Katz: Yeah, that’s really interesting, and it also indicates the non-obvious consequences of the way that the FDA regulates models and how that influences the architecture these models take on. I want to get to the predetermined change control plan and how you navigate that, but before we do, can you talk about the unsupervised image quality enhancement you’re doing to normalize across these different sensor distributions? Can you explain what that problem looks like, why you did it, and how you approached it?
Sadegh Salehi: Yeah, that is a very interesting one. In the dental industry, you have different sensors, and each of these sensors most of the time has different acquisition software. And then you have the visualization for these different software, the PMS one. The dentist looks at the image and diagnoses and introduces the treatment plan. What happens is that each of these acquisition software systems has their own post-processing on top of the raw data to make the image maybe sharper, less noisy—all of that post-processing after the image acquisition. But this post-processing is designed for one sensor, one image acquisition. That’s it. It’s very specialized work from their company. Now in Overjet, we have access to that raw data, but we don’t have access to that post-processed data most of the time. We want to make sure that we have a high-quality image on top of that raw data across all of these different sensors and different acquisition systems. The problem is that you don’t have access to the good quality image—just the raw image—and you cannot even annotate those images. How do you annotate image quality? Someone needs to paint on it; you cannot annotate that. So the way that we trained this unsupervised model—it has an internal architecture, we haven’t released this architecture, we haven’t publicly published it, but it’s a good one—the idea behind this image enhancement was that we trained the model in an unsupervised manner. We trained the model to learn artifacts rather than learn image content. We trained the model to learn what the artifact is, what is the distribution of noise, what is the distribution of the artifacts you will observe on these different datasets, what is the different distribution of contrast artifacts. It learns those artifacts and it’s a residual model. It removes the artifact from the raw image. It is not outputting the processed image; it is outputting the artifact image, and then we subtract that artifact from the raw image.
Ross Katz: So Overjet operates under FDA regulation, which means, as you mentioned earlier, you can’t just retrain a model and push it to production. Can you explain the PCCP and how it changes the way you have to think about updating AI models?
Sadegh Salehi: What happens in regulated space and in the medical imaging domain is that before the deep learning era—2017, 2018—people had their traditional image processing algorithms. Most of them were not learning-based, you do all those things and you get the clearance. You are not going to change the whole pipeline in the next 10 years. But after deep learning came, people realized, “I train my data, I train my model, I get the clearance. But now I can train a better model in three months. Do I need to go through all this comprehensive evaluation and send it to the regulated space and wait three months?” It doesn’t make sense. And to be fair, I think the FDA is following the tech, and they changed the rules in a way that makes sense. So in 2020 or 2021—if I’m not mistaken—FDA introduced this predetermined change control plan where you add this into your filing and you are saying, “This is my filing, this is my evaluation of this model. Also, I have this predetermined change that I am going to apply.” Which means, “I am going to retrain this model with 10 percent more data, but I’m not going to change the intended use of this device, I’m not going to change the input of this device.” Which means that if this device is cleared for bitewing images, I’m not going to change it so that it is working for pano images—I’m just making that up. But I’m going to retrain this model with this extra data and I’m going to do a comprehensive evaluation. I am going to make sure that it is passing all these predefined thresholds. I am going to test this on all these predefined subgroups. And if I pass everything, I don’t need to file another clearance to put these new weights into my product. I put it on the shelf, you can audit me, you can come and ask me what those changes were. And then you send it to the FDA during the filing and the FDA says, “Yeah, okay, good. I agree with your predetermined change plan. Now you don’t need to send every single change to the FDA.” Overjet was one of the very first companies that got clearance with PCCP. And that makes our life way easier now for the indications where we have PCCP. We just retrain a model, we follow our comprehensive evaluation protocols, and when we pass all of those things, we just put it in the product without going through the regulated space again.
Ross Katz: Yeah, that’s really interesting, and it makes a lot of sense both from your perspective and from the FDA’s perspective. Since you were one of the first, my understanding is that you have this staged deployment process where you start with a shadow model—the model is doing inference but not actually making predictions in production—and then you have a canary where it’s only serving a small percentage of traffic, then an A/B testing phase, and then full production. Is that how it works? And if so, I’m interested in how you monitor and connect to what’s happening in production.
Sadegh Salehi: Monitoring is really interesting for us because we are very lucky—we have some sort of label in production. It’s a noisy label, but let me tell you what I mean. In dentistry, what happens is that the patient goes to the office, they take the X-ray image, then the dentist examines the tooth. And then they provide the treatment plan. They say, “Okay, I’m going to put the filling on your tooth number two.” The patient accepts that treatment plan and they do the procedure. Now that treatment plan has different procedure codes, and we have access to that treatment plan. That’s a very important part of this puzzle. This treatment plan is happening after the image acquisition. So you have the image, you have the treatment plan. What does that mean? If there is a treatment plan with a procedure code of filling on the mesial side of tooth number two, that means there was something wrong on the mesial side of tooth number two, and most likely it is cavity because they do the filling for the cavity. Which means that if our model detects that cavity, we can say it is a true positive. We detect it, the doctor applied that treatment. Why do I say it is noisy? Because assume there is no treatment plan for this one, but we detect the cavity for the mesial side of tooth number two. Why is there no treatment plan? Because sometimes there is a cavity, but the cavity is too small. They don’t want to do the filling. They say, “Okay, there is a fluoride treatment you can do and you’re good.” Or sometimes we don’t detect anything, and they do the treatment. Why? Is it a false negative? Not necessarily, because sometimes that decay is not visible in the X-ray. They did the examination and they saw that the decay is on the occlusal side of the tooth, and if the decay is on the occlusal side of the tooth, you cannot see it in the X-ray. So these are noisy labels, but you can sample these things so you can get a feeling of whether your model is doing well, because you have some sort of information afterward. That’s one of our main monitoring systems.
Ross Katz: Yeah, that’s really interesting. If you’re in that shadow model or canary model phase, you get both the ground truth label from the clinic and the predictions of both models. But there’s also the feedback loop between the model you have in production and the clinic itself. Because if the model that’s in production didn’t predict that the tooth decay was there, then the clinic might not notice it. But maybe your new model is so much better that it’s noticing the tooth decay, and if that model were in production, there would have been a treatment plan in the first place. Exactly. So certainly it’s not as easy as it may seem on the surface, but as long as you’re being rigorous about your evaluations and you’ve created that plan up front with the FDA, then you can be reasonably confident that the model you’re rolling out is as good or better than the one that you had in place before. Exactly. Well, Sadegh, it’s been awesome having you on the podcast. I really appreciate it, and I look forward to connecting down the line.
Sadegh Salehi: Yeah, thank you for having me. It was fun. I really enjoyed it.
Ross Katz: Sadegh, thank you. That was Sadegh Salehi, Director of Research and Principal Scientist at Overjet. So here’s what I’m taking away from this one. The subgroup problem is real, and it’s sneaky. You get an F1 of 80 percent and you think you’ve got a strong model. But Sadegh made the point that 80 percent is a meaningless number if you don’t know what it’s 80 percent across. He walked through an example where they trained a decay model mostly on molars. These are thick teeth, dense; the decay shows up as a subtle gray shade on the X-ray. And then they ran it on anterior teeth—thinner teeth that naturally show more gray. The model flagged everything. Huge false positive rates on a whole category of teeth. And the aggregate metric just—it didn’t surface that. It was buried. And that’s why Sadegh asks his team to provide not just one number to evaluate models, but a variety of numbers that sort of do that cross-sectional analysis. And then the shortcut thing, which I thought was one of the sharpest moments in the conversation. If all the images from a rare sensor in your training data happen to be healthy, the model doesn’t learn how to read images from that sensor. It just learns that the sensor means healthy. It’s not even looking at the teeth anymore; it found an easier pattern. And if you’re evaluating aggregate metrics, you’ll never catch that. Sadegh’s point—and I think this is the thesis of the whole episode—is that the same flawed sampling that creates gaps in your training data also creates them in your test data. So if your evaluation isn’t catching your blind spots, then it’s obviously confirming them. The other thread I keep thinking about is the architecture story. They started with 20-something narrow models, each one trained on thousands of images for one condition, one image type, one sensor. As the data grew to millions, they used those small models to generate noisy labels, and then they used those noisy labels as the training signal for a much larger foundation model that they call Unity. And then they built independent prediction heads on top, one per condition, specifically so that they could retrain one head without having to reclear every other head with the FDA. That’s not just a modeling decision; that’s a regulatory decision that they made at the architectural level, and I thought that was really smart and interesting. And then there’s the monitoring piece. Overjet has access to treatment plan codes, the CDT codes that he mentioned, that get assigned after a clinician examines a patient and actually performs a procedure. So if a filling goes on a tooth where the model didn’t flag decay, that’s a signal. It’s noisy—maybe the decay was too small to film, maybe it was on a surface X-rays can’t see—but it’s a real feedback loop between what the model predicted and what the dentist actually did. Most teams in production ML don’t have the benefit of something like that. Whether you’re in dental imaging or building models in a completely different domain, I think the core points hold. If you’re not stratifying your evaluation by the dimensions that actually matter when your model hits production, then your numbers are telling you a story that might not be the right one. Sadegh is on LinkedIn—the links in the show notes. If you want to see what Overjet is building, then go to Overjet.com. And if you got something out of this, then please share it with someone who’s building models against messy data. They’ll thank you. I’m Ross Katz, thanks for listening.
Jason: And that’s it for this episode of Data in Biotech. If you enjoyed the episode, please subscribe, rate, or leave a review in your podcast player of choice. See you next time.
