Medical AI · Clinical model validation
We design and run validation for clinical prediction models and medical AI: risk scores, diagnostic classifiers and imaging models. The goal is evidence that holds on patients, sites and time periods the model never saw, reported so a clinical reviewer can follow it.
Where validation goes wrong
Optimistic results usually come from the evaluation, not the model. These are the problems we check for first.
Validation is part of our Medical AI practice, and runs through every project we take on.
Information about the outcome reaches the inputs. Common sources: features recorded after the prediction time, codes entered because of the diagnosis, preprocessing fitted on all the data, and the same patient in training and test. Each one inflates performance without warning.
Visits, records and images from one patient are correlated. Split by record, and the model can recognise patients instead of learning the disease. We split by patient, and by site or time when the question is whether the model transfers.
A new hospital brings different patients, prevalence, devices, coding habits and workflows. Performance can drop, and calibration often drifts first. Cross-validation on the development data cannot measure this. Only data from elsewhere can.
Models can learn markers of the hospital, the device or the care pathway that happen to correlate with the outcome. Such a model scores well where it was built and fails elsewhere. Per-site results and error review expose it.
Internal, temporal & external
Each level of validation answers a different question. We plan which ones the data can support before any modelling starts.
Cross-validation or bootstrapping on the development data, with every preprocessing and selection step repeated inside each fold. It estimates and corrects optimism. It does not show that the model transfers.
Train on earlier patients, test on later ones. It catches drift in practice, coding and population, and mirrors real use: the model will only ever see patients who arrive after it was built.
Test the frozen model on another site, region or health system. In our own published work, diabetes risk scores were developed on UK Biobank data, and one was externally validated on Clalit Health Services records.
Edlitz & Segal, eLife 2022 →Too few outcome events make both development and validation unstable. We check the data against published sample-size methods for prediction models before promising a result, and give confidence intervals for every metric.
Riley et al., BMJ 2020 →What we measure
A model can rank patients well and still give the wrong risk. We report several measures, each with uncertainty, against criteria agreed in advance.
AUROC (the C-statistic) for ranking, plus precision-recall curves when the outcome is rare. Sensitivity, specificity, PPV and NPV at the thresholds the clinic would actually use, since predictive values change with prevalence.
Calibration plots, calibration slope and calibration-in-the-large. A predicted risk of 20% should mean about 20 in 100 similar patients have the event. A miscalibrated score misleads treatment thresholds even when its ranking is good.
Van Calster et al., BMC Medicine 2019 →Decision curve analysis compares acting on the model with treating everyone or no one, across a range of thresholds. It asks whether using the model would do more good than harm, not only whether it is accurate.
Vickers & Elkin, 2006 →Performance and calibration by site, age, sex and other groups that matter for the intended use. Intervals show when a subgroup is too small to judge. Gaps are reported, not averaged away.
We do not quote target AUCs before seeing the data. Success criteria are set with your clinical team against the intended use.
Reporting standards
Published guidelines set out what a prediction-model study must disclose. We structure validation reports around them, so the items a reviewer looks for are there.
The 2024 reporting guideline for clinical prediction models that use regression or machine learning. It covers data sources, outcome definitions, sample size, missing data, performance measures and fairness.
Collins et al., BMJ 2024 →A tool for judging the quality, risk of bias and applicability of prediction-model studies. We use it as a self-check on our own study design, before anyone else applies it.
Moons et al., BMJ 2025 →DECIDE-AI covers the early clinical evaluation of AI decision support, such as a silent-mode phase. CLAIM covers studies of AI in medical imaging. We follow whichever fits the study.
DECIDE-AI, Nature Medicine 2022 →MLAIA produces validation evidence: study design, analysis, results and documented limitations. We are not regulatory consultants. Regulatory strategy and submissions stay with your regulatory team or advisers, and engaging MLAIA does not imply any regulatory status for your product.
How an engagement runs
Most validation problems are decided before any model is trained: by the prediction time, the outcome definition and the split. Our process starts there.
Intended use, prediction time, outcome definition, population and success criteria. We fix the split strategy and lock a test set before any model sees it.
Trace every feature to when it is recorded relative to the prediction time. Check for repeated patients, label leakage, missing-data patterns and differences between sites.
Internal, temporal and external validation as the data allows. Discrimination, calibration, clinical usefulness and subgroups, each with confidence intervals and known failure modes.
A validation report structured around TRIPOD+AI, with limitations stated plainly, plus code your team can rerun. Where useful, a plan for prospective or silent-mode evaluation.
This page goes deeper on the retrospective validation step of our medical engagement model.
Sources
Before we start
Internal validation tests the model on data from the same source it was built on, using cross-validation or bootstrapping. It estimates optimism. External validation tests the frozen model on patients from other sites or later periods. Only external validation shows whether the model transfers.
Common causes are leakage in the original evaluation, splitting by record instead of by patient, and real shift in patients, prevalence or measurement at the new site. We audit the original evaluation first, because a leak changes what the drop means.
No. AUC measures ranking only. A model can rank well and still over- or underestimate risk, which matters when decisions use a threshold. We also report calibration, measures at the clinical threshold and decision curves.
Often, yes. If ranking holds but calibration drifts, updating the intercept or slope on local data can fix much of the problem. If ranking drops too, the model may need refitting. Either way, the updated model needs its own validation.
Yes. Independent validation of a vendor or in-house model is a common need. We need the model's outputs on your data, or the model itself, and a clear statement of its intended use.
No. We produce the validation evidence that regulatory work relies on: study design, results with uncertainty, subgroup analysis and documented limitations. We are not regulatory consultants, and engaging MLAIA does not imply any regulatory status for your product.
Keep exploring
Medical AI · Clinical Model Validation
Tell us the clinical question, the model or data you have and what a reviewer would need to see. We’ll tell you whether it’s feasible and what the first step would be.
Please don’t include confidential datasets, credentials or patient information.