Medical AI lab · Patient leakage
Patient-level data leakage: interactive demo
Patient-level data leakage happens when records from the same patient end up in both the training and the test set. The model can then score well by recognising the patient rather than learning the outcome, and the test result overstates how it will do on new patients. In this demo the same model is tested on synthetic visits split by record and split by patient, and both results are compared with its AUC on patients it has never seen.
These demos are illustrative. Every signal, image, patient and note is synthetic and fictional, generated and scored in your browser, and every number shown is computed from that synthetic data. They show general principles, not the performance of any real model, product or project, and none of them is a medical device.
Nothing is uploaded. No patient data is used.
Clinical model validation / data splits
Same model. Two test scores.
What this shows: split by record, a patient’s other visits sit in the training set, and the model scores well by recognising patients. Split by patient, the test score matches what it does on new patients.
Split the data
300 synthetic patients, 30% with the outcome; 25% of records, or of patients, go to the test set. The model is a 7-nearest-neighbour classifier on 8 features. Try one visit per patient: the two splits then agree.
Who is in the test set
first 12 of 300 patientsGrey: training recordBlue: test record
Bars start at AUC 0.5, chance level. New patients: 240 synthetic patients the model never saw.
Synthetic data · illustrative · not a medical device
Any clinical dataset with repeated visits, admissions, images or samples per patient. The split has to follow the unit the model will meet in practice: new patients, and often new sites or later time periods.
What this experiment models
Each synthetic patient has a stable set of 8 features, a fingerprint, and a single outcome that shifts 3 of those features slightly. Each visit adds small visit-to-visit noise to the fingerprint, so a patient’s visits are much more alike than two different patients. A distance-weighted 7-nearest-neighbour classifier is scored with the area under the ROC curve (AUC). The data is fixed by a seed, so the same settings always give the same numbers.
The principle
How it works
Each of the 300 synthetic patients has a stable set of eight features and one outcome, which shifts three of the features slightly. Every visit repeats the patient’s features with a little noise, so two visits of one patient are far more alike than two patients with the same outcome. Real records behave the same way: age, chronic conditions and baseline measurements carry over from visit to visit.
The model is a k-nearest-neighbour classifier, which predicts from the most similar training records. Split by record, a test visit’s nearest neighbours are usually the same patient’s other visits, which carry the right label, so the test AUC climbs towards 1 as visits per patient increase. Split by patient, every test patient is new to the model, and the test AUC stays close to the AUC on 240 fresh patients. With one visit per patient the two splits are equivalent.
Flexible models such as gradient boosting and neural networks can memorise patients in the same way. The fix is in the evaluation design: split by patient, and by site or time period where the model will meet new ones. See clinical model validation.
Medical AI lab
More demos
All six demos on one page: the medical AI labClinical model validation →