Medical AI lab · Calibration & thresholds
Calibration and decision thresholds: interactive demo
Calibration is whether a model’s predicted risks match what actually happens: of patients given a 20% risk, about 20% should have the outcome. A model can rank patients well, with a high AUC, and still be badly calibrated. This demo shows a risk model’s calibration plot and decision curve, and counts missed cases and false alarms per 1,000 patients, as you move the treatment threshold, the prevalence of the disease and the model’s calibration.
These demos are illustrative. Every signal, image, patient and note is synthetic and fictional, generated and scored in your browser, and every number shown is computed from that synthetic data. They show general principles, not the performance of any real model, product or project, and none of them is a medical device.
Nothing is uploaded. No patient data is used.
Clinical model validation / calibration
Same ranking. Different usefulness.
What this shows: a model that ranks patients just as well (same AUC) can still give wrong risks, flag the wrong number of people and deliver less net benefit, especially at a hospital where the disease is more or less common.
Set the decision
The model was developed where 10% of patients had the outcome. Move the prevalence to deploy it somewhere else.
Calibration and decision curve
4,000 synthetic patientsBlue: modelAmber: treat everyoneGrey: treat no oneWhite dashed: perfect calibration, threshold
Synthetic data · illustrative · not a medical device
Risk scores that trigger an action: a test, a referral, a treatment. Discrimination alone does not show whether the threshold makes sense, and a model moved to a new population usually needs its calibration checked again.
What this experiment models
4,000 synthetic patients with a risk score. True risk is the logistic function of the score plus an intercept set by the prevalence slider. The model is the same logistic function with the intercept that fits 10% prevalence; the miscalibrated variant multiplies its log-odds by 0.6 and adds 1, so most patients get too high a risk and the highest-risk patients too low a one. Outcomes are expected values, not random draws, so the numbers do not jitter. Net benefit follows decision-curve analysis: true positives minus false positives weighted by the threshold odds, t / (1 − t), per patient.
The principle
How it works
The 4,000 synthetic patients each have a risk score, and their true risk follows from it. The model was developed where 10% of patients had the outcome, and there it is perfectly calibrated. Move the prevalence slider and the same model is deployed in a hospital with more or less disease: the ranking barely changes, but every predicted risk is now too low or too high. The miscalibrated toggle compresses and shifts the model’s risks; it is a monotone change, so the AUC stays exactly the same.
The threshold turns risks into decisions. Every patient at or above it is treated or referred, which sets the missed cases and false alarms. Decision-curve analysis weighs the two using the threshold itself: choosing 10% means one case found is worth nine unnecessary treatments. Net benefit is the true-positive rate minus the false-positive rate times t / (1 − t), compared with treating everyone and treating no one.
That is why a clinical reviewer asks for calibration and clinical usefulness, not only discrimination, and asks again when the model moves to a new population. See clinical model validation.
Medical AI lab
More demos
All six demos on one page: the medical AI labClinical model validation →