Skip to the medical AI lab

The medical AI lab

Interactive Medical AI Lab

Six hands-on experiments in how medical AI is checked: a heart-rate estimate that looks right while its beats are wrong, a test score inflated by patient leakage, risk thresholds and calibration, an imaging model moved to a new scanner, a Dice score that hides a missed lesion, and a clinical note that is only partly de-identified.

SYNTHETIC DATA

These demos are illustrative. Every signal, image, patient and note is synthetic and fictional, generated and scored in your browser, and every number shown is computed from that synthetic data. They show general principles, not the performance of any real model, product or project, and none of them is a medical device.

Nothing is uploaded. No patient data is used.

01

Biosignals / clinical model validation

Right on average. Wrong beat by beat.

What this shows: as motion artifact grows, a beat detector misses real beats and adds false ones, and the two errors can cancel so the average heart rate still looks right.

Open as a page: Heart rate from a noisy ECG →

Add motion artifact

You can also drag across the trace, or focus it and use the arrow keys.

The check passes when the estimated rate is within ±5 bpm of the true rate and at most one beat is false or missed. Watch for “fail: beats” with a small rate error.

Synthetic ECG and detected beats

sliding window · 250 Hz

Heart rate in the visible window

true HR · bpm
72.0
estimated HR · bpm
72.0
error
±0.0 bpm
detected / actual
9 / 9
false · missed
0 · 0
validation check
pass

Amber: ECG with motion artifactBlue: detected beats, detector energyWhite: true beats, threshold

Synthetic data · illustrative · not a medical device

Where this matters

Wearables, patient monitors and any algorithm that reports a rate or count from a biosignal. A validation that compares only averages can pass a detector whose individual beats are wrong.

What this experiment models

A synthetic ECG built from Gaussian P, QRS and T waves at a known, slowly drifting heart rate, plus motion artifact made of baseline wander, muscle noise and movement bursts. The detector is a classical QRS detector: zero-phase 5–15 Hz band-pass, derivative, squaring, a 150 ms moving window, an adaptive threshold and a 250 ms refractory period. Detected beats are matched to true beats within 100 ms. It is the same code as the live demo on the Medical AI page.

02

Clinical model validation / data splits

Same model. Two test scores.

What this shows: split by record, a patient’s other visits sit in the training set, and the model scores well by recognising patients. Split by patient, the test score matches what it does on new patients.

Open as a page: Patient-level data leakage →

Split the data

Assign to the test set

300 synthetic patients, 30% with the outcome; 25% of records, or of patients, go to the test set. The model is a 7-nearest-neighbour classifier on 8 features. Try one visit per patient: the two splits then agree.

Who is in the test set

first 12 of 300 patients

Grey: training recordBlue: test record

Bars start at AUC 0.5, chance level. New patients: 240 synthetic patients the model never saw.

Reported test AUC—
AUC on new patients—
Optimism—

Synthetic data · illustrative · not a medical device

Where this matters

Any clinical dataset with repeated visits, admissions, images or samples per patient. The split has to follow the unit the model will meet in practice: new patients, and often new sites or later time periods.

What this experiment models

Each synthetic patient has a stable set of 8 features, a fingerprint, and a single outcome that shifts 3 of those features slightly. Each visit adds small visit-to-visit noise to the fingerprint, so a patient’s visits are much more alike than two different patients. A distance-weighted 7-nearest-neighbour classifier is scored with the area under the ROC curve (AUC). The data is fixed by a seed, so the same settings always give the same numbers.

03

Clinical model validation / calibration

Same ranking. Different usefulness.

What this shows: a model that ranks patients just as well (same AUC) can still give wrong risks, flag the wrong number of people and deliver less net benefit, especially at a hospital where the disease is more or less common.

Open as a page: Calibration and decision thresholds →

Set the decision

The model was developed where 10% of patients had the outcome. Move the prevalence to deploy it somewhere else.

Calibration and decision curve

4,000 synthetic patients
Calibration: predicted vs observed, 10 risk groups
Calibration plotPredicted against observed risk.
Decision curve: net benefit vs threshold
Decision curveNet benefit of the model, treat all and treat none.

Blue: modelAmber: treat everyoneGrey: treat no oneWhite dashed: perfect calibration, threshold

AUC—
Mean risk: predicted / observed—
Net benefit per 1,000—
Missed cases per 1,000—
False alarms per 1,000—
Flagged per 1,000—

Synthetic data · illustrative · not a medical device

Where this matters

Risk scores that trigger an action: a test, a referral, a treatment. Discrimination alone does not show whether the threshold makes sense, and a model moved to a new population usually needs its calibration checked again.

What this experiment models

4,000 synthetic patients with a risk score. True risk is the logistic function of the score plus an intercept set by the prevalence slider. The model is the same logistic function with the intercept that fits 10% prevalence; the miscalibrated variant multiplies its log-odds by 0.6 and adds 1, so most patients get too high a risk and the highest-risk patients too low a one. Outcomes are expected values, not random draws, so the numbers do not jitter. Net benefit follows decision-curve analysis: true positives minus false positives weighted by the threshold odds, t / (1 − t), per patient.

04

Medical imaging AI / domain shift

Tuned on one scanner. Tested on another.

What this shows: a segmentation rule tuned on scanner A fails on scanner B when contrast or noise changes, although the anatomy is identical. Normalising intensities first removes most of the contrast effect, not the noise.

Open as a page: Scanner shift in medical imaging →

Change scanner B

The model smooths the image and keeps pixels inside an intensity window. The window was chosen to give the best Dice on scanner A.

Same anatomy, two scanners

128 × 128 synthetic slice
Scanner A · where the model was tuned
Scanner B · new site

Blue: correct lesion pixelsRed: false lesionAmber: missed lesion

Scanner B intensity histogramLesion and tissue intensities and the model’s window.

Blue: lesion pixels on BAmber: other tissue on BWhite dashed: window learned on A

Dice on scanner A—
Dice on scanner B—
Extra · missed pixels on B—

Synthetic data · illustrative · not a medical device

Where this matters

Imaging models moved between scanners, protocols, vendors or sites. Learned models are far more flexible than this rule, but they can still depend on intensity and noise characteristics of the scanners they were trained on, which is why external, multi-site testing matters.

What this experiment models

A procedural slice: a body outline, two organs, a bright bony structure and an irregular lesion with a brighter core. Scanner A adds fixed Gaussian noise. Scanner B scales contrast around the image mean and adds its own noise. The “model” is a 3 × 3 mean filter followed by an intensity window, grid-searched for the best Dice on scanner A and then frozen. Normalisation, when on, z-scores each image before the window is applied, and the window is re-tuned on normalised scanner A.

05

Medical imaging AI / evaluation metrics

A high Dice. A missed lesion.

What this shows: Dice is dominated by the largest structure, so missing a small lesion barely moves it. A boundary distance and a simple count of lesions found tell a different story.

Open as a page: Dice score vs clinical error →

Change the prediction

Two true lesions: one about 68 mm across and one small. Shrink the small lesion below about 5 mm and even the 95th-percentile distance stops noticing that it was missed.

Ground truth and prediction

160 × 160 mm · 1 mm pixels

Blue: predicted and trueAmber: missedRed: predicted, not true

Dice—
HD95 boundary distance—
Max boundary distance—
Lesions detected—

Synthetic data · illustrative · not a medical device

Where this matters

Segmentation and detection where small findings change the clinical decision: metastases, small nodules, early lesions. Report per-lesion detection and a boundary metric next to Dice, and choose metrics for the clinical question.

What this experiment models

Binary masks on a 160 × 160 grid with 1 mm pixels. The large true lesion is a slightly irregular disc; the prediction follows its boundary with a fixed wobble scaled by the roughness slider. Dice is 2|A∩B| / (|A| + |B|) over all pixels. Boundary distances are measured between the boundary pixels of the two masks in both directions; HD95 is the larger of the two directed 95th percentiles, and the maximum is the Hausdorff distance. A lesion counts as detected if any predicted pixel overlaps it.

06

Clinical data de-identification / free text

Names are easy. The rest re-identifies.

What this shows: simple rules catch titled names and ID numbers early; dates, places and rare details such as an unusual job or a unique event need more, and turning the rules up also removes doses, drug names and ordinary words.

Open as a page: De-identifying a clinical note →

Run the detector

Note language
View

Rules switched on

    Fictional discharge note

    13 annotated identifiers

    Green: identifier caughtRed: identifier missedAmber: over-redacted

    Identifiers caught—
    Missed—
    Over-redacted—
    Caught by type

    Synthetic data · fictional note · illustrative · not a medical device

    Where this matters

    Free-text notes, letters and reports before they reach analysts, vendors or language models. Measure missed identifiers and over-redaction on your own notes and languages; de-identification lowers disclosure risk but is not a guarantee of anonymity.

    What this experiment models

    One fictional note in three versions, with a ground-truth annotation of 13 identifiers: names, dates, an ID number and a phone number (all zeros), an address and towns, and two rare details. The detector is a list of regular expressions and word lists, switched on in five levels. An identifier counts as caught when at least half of its characters are redacted; a redaction that touches no identifier counts as over-redaction. Real systems combine rules with trained models and human review.

    MLAIA DATA SCIENCE

    Tell us what
    you’re working on.

    Share the problem, the data you have and what success would look like. We’ll discuss a practical next step.

    Please don’t include confidential datasets, credentials or patient information.

    yochai@mlaia.com
    WhatsApp: +972 52 484 6282

    Your inquiry is handled under our Privacy Policy.