The medical AI lab
Interactive Medical AI Lab
Six hands-on experiments in how medical AI is checked: a heart-rate estimate that looks right while its beats are wrong, a test score inflated by patient leakage, risk thresholds and calibration, an imaging model moved to a new scanner, a Dice score that hides a missed lesion, and a clinical note that is only partly de-identified.
These demos are illustrative. Every signal, image, patient and note is synthetic and fictional, generated and scored in your browser, and every number shown is computed from that synthetic data. They show general principles, not the performance of any real model, product or project, and none of them is a medical device.
Nothing is uploaded. No patient data is used.
Biosignals / clinical model validation
Right on average. Wrong beat by beat.
What this shows: as motion artifact grows, a beat detector misses real beats and adds false ones, and the two errors can cancel so the average heart rate still looks right.
Open as a page: Heart rate from a noisy ECG →Add motion artifact
You can also drag across the trace, or focus it and use the arrow keys.
The check passes when the estimated rate is within ±5 bpm of the true rate and at most one beat is false or missed. Watch for “fail: beats” with a small rate error.
Synthetic ECG and detected beats
sliding window · 250 HzHeart rate in the visible window
- true HR · bpm
- 72.0
- estimated HR · bpm
- 72.0
- error
- ±0.0 bpm
- detected / actual
- 9 / 9
- false · missed
- 0 · 0
- validation check
- pass
Amber: ECG with motion artifactBlue: detected beats, detector energyWhite: true beats, threshold
Synthetic data · illustrative · not a medical device
Wearables, patient monitors and any algorithm that reports a rate or count from a biosignal. A validation that compares only averages can pass a detector whose individual beats are wrong.
What this experiment models
A synthetic ECG built from Gaussian P, QRS and T waves at a known, slowly drifting heart rate, plus motion artifact made of baseline wander, muscle noise and movement bursts. The detector is a classical QRS detector: zero-phase 5–15 Hz band-pass, derivative, squaring, a 150 ms moving window, an adaptive threshold and a 250 ms refractory period. Detected beats are matched to true beats within 100 ms. It is the same code as the live demo on the Medical AI page.
Clinical model validation / data splits
Same model. Two test scores.
What this shows: split by record, a patient’s other visits sit in the training set, and the model scores well by recognising patients. Split by patient, the test score matches what it does on new patients.
Open as a page: Patient-level data leakage →Split the data
300 synthetic patients, 30% with the outcome; 25% of records, or of patients, go to the test set. The model is a 7-nearest-neighbour classifier on 8 features. Try one visit per patient: the two splits then agree.
Who is in the test set
first 12 of 300 patientsGrey: training recordBlue: test record
Bars start at AUC 0.5, chance level. New patients: 240 synthetic patients the model never saw.
Synthetic data · illustrative · not a medical device
Any clinical dataset with repeated visits, admissions, images or samples per patient. The split has to follow the unit the model will meet in practice: new patients, and often new sites or later time periods.
What this experiment models
Each synthetic patient has a stable set of 8 features, a fingerprint, and a single outcome that shifts 3 of those features slightly. Each visit adds small visit-to-visit noise to the fingerprint, so a patient’s visits are much more alike than two different patients. A distance-weighted 7-nearest-neighbour classifier is scored with the area under the ROC curve (AUC). The data is fixed by a seed, so the same settings always give the same numbers.
Clinical model validation / calibration
Same ranking. Different usefulness.
What this shows: a model that ranks patients just as well (same AUC) can still give wrong risks, flag the wrong number of people and deliver less net benefit, especially at a hospital where the disease is more or less common.
Open as a page: Calibration and decision thresholds →Set the decision
The model was developed where 10% of patients had the outcome. Move the prevalence to deploy it somewhere else.
Calibration and decision curve
4,000 synthetic patientsBlue: modelAmber: treat everyoneGrey: treat no oneWhite dashed: perfect calibration, threshold
Synthetic data · illustrative · not a medical device
Risk scores that trigger an action: a test, a referral, a treatment. Discrimination alone does not show whether the threshold makes sense, and a model moved to a new population usually needs its calibration checked again.
What this experiment models
4,000 synthetic patients with a risk score. True risk is the logistic function of the score plus an intercept set by the prevalence slider. The model is the same logistic function with the intercept that fits 10% prevalence; the miscalibrated variant multiplies its log-odds by 0.6 and adds 1, so most patients get too high a risk and the highest-risk patients too low a one. Outcomes are expected values, not random draws, so the numbers do not jitter. Net benefit follows decision-curve analysis: true positives minus false positives weighted by the threshold odds, t / (1 − t), per patient.
Medical imaging AI / domain shift
Tuned on one scanner. Tested on another.
What this shows: a segmentation rule tuned on scanner A fails on scanner B when contrast or noise changes, although the anatomy is identical. Normalising intensities first removes most of the contrast effect, not the noise.
Open as a page: Scanner shift in medical imaging →Change scanner B
The model smooths the image and keeps pixels inside an intensity window. The window was chosen to give the best Dice on scanner A.
Same anatomy, two scanners
128 × 128 synthetic sliceBlue: correct lesion pixelsRed: false lesionAmber: missed lesion
Blue: lesion pixels on BAmber: other tissue on BWhite dashed: window learned on A
Synthetic data · illustrative · not a medical device
Imaging models moved between scanners, protocols, vendors or sites. Learned models are far more flexible than this rule, but they can still depend on intensity and noise characteristics of the scanners they were trained on, which is why external, multi-site testing matters.
What this experiment models
A procedural slice: a body outline, two organs, a bright bony structure and an irregular lesion with a brighter core. Scanner A adds fixed Gaussian noise. Scanner B scales contrast around the image mean and adds its own noise. The “model” is a 3 × 3 mean filter followed by an intensity window, grid-searched for the best Dice on scanner A and then frozen. Normalisation, when on, z-scores each image before the window is applied, and the window is re-tuned on normalised scanner A.
Medical imaging AI / evaluation metrics
A high Dice. A missed lesion.
What this shows: Dice is dominated by the largest structure, so missing a small lesion barely moves it. A boundary distance and a simple count of lesions found tell a different story.
Open as a page: Dice score vs clinical error →Change the prediction
Two true lesions: one about 68 mm across and one small. Shrink the small lesion below about 5 mm and even the 95th-percentile distance stops noticing that it was missed.
Ground truth and prediction
160 × 160 mm · 1 mm pixelsBlue: predicted and trueAmber: missedRed: predicted, not true
Synthetic data · illustrative · not a medical device
Segmentation and detection where small findings change the clinical decision: metastases, small nodules, early lesions. Report per-lesion detection and a boundary metric next to Dice, and choose metrics for the clinical question.
What this experiment models
Binary masks on a 160 × 160 grid with 1 mm pixels. The large true lesion is a slightly irregular disc; the prediction follows its boundary with a fixed wobble scaled by the roughness slider. Dice is 2|A∩B| / (|A| + |B|) over all pixels. Boundary distances are measured between the boundary pixels of the two masks in both directions; HD95 is the larger of the two directed 95th percentiles, and the maximum is the Hausdorff distance. A lesion counts as detected if any predicted pixel overlaps it.
Clinical data de-identification / free text
Names are easy. The rest re-identifies.
What this shows: simple rules catch titled names and ID numbers early; dates, places and rare details such as an unusual job or a unique event need more, and turning the rules up also removes doses, drug names and ordinary words.
Open as a page: De-identifying a clinical note →Run the detector
Rules switched on
Fictional discharge note
13 annotated identifiersGreen: identifier caughtRed: identifier missedAmber: over-redacted
Synthetic data · fictional note · illustrative · not a medical device
Free-text notes, letters and reports before they reach analysts, vendors or language models. Measure missed identifiers and over-redaction on your own notes and languages; de-identification lowers disclosure risk but is not a guarantee of anonymity.
What this experiment models
One fictional note in three versions, with a ground-truth annotation of 13 identifiers: names, dates, an ID number and a phone number (all zeros), an address and towns, and two rare details. The detector is a list of regular expressions and word lists, switched on in five levels. An identifier counts as caught when at least half of its characters are redacted; a redaction that touches no identifier counts as over-redaction. Real systems combine rules with trained models and human review.
One demo per page
Open a demo on its own
Each experiment also has its own page, with a short explanation of the principle behind it.
- 01Heart rate from a noisy ECG
- 02Patient-level data leakage
- 03Calibration and decision thresholds
- 04Scanner shift in medical imaging
- 05Dice score vs clinical error
- 06De-identifying a clinical note
Clinical model validation →Medical imaging AI →Clinical data de-identification →