Sample size for model validation: interactive demo

Sensitivity is the share of patients with the condition that the model detects, so it is measured only on the patients who have the condition. In a validation set of 200 patients with 10% prevalence that is 20 people. Specificity is measured on the other 180. The two estimates come from the same study and have very different precision.

The demo draws the 95% Wilson score interval around the measured sensitivity, which is a standard interval for a proportion and behaves well with small counts and with values near 100%. The grey bar shows the same measured value on the negatives, and the white dots show 60 simulated repeats of the study with a model whose true sensitivity is exactly the value entered. The spread of the dots is what the interval describes: the same model, tested again on a new sample of the same size, gives a different number.

The amber line is the minimum sensitivity you require. A study supports the claim only when the whole interval is above that line, and the measured value alone does not show it. The useful planning question is therefore how many positive cases are needed for the interval to be narrow enough, and then how many patients must be collected to get them at the expected prevalence. See clinical model validation.

Picture a team validating a model that flags a condition present in about one patient in ten. They collect 200 consecutive patients, run the model and find that it detects 90% of the patients with the condition. The requirement was at least 80%, so the result goes into the report as a pass.

The demo’s default settings are that study: 200 patients, 10% prevalence, an expected sensitivity of 90% and a required minimum of 80%. Positive cases reads 20, and the measured sensitivity is 18 of 20. The 95% interval is 70% to 97%. The lower end is below the 80% requirement, the bar turns red, and the demo’s verdict is that this study cannot show the model is good enough.

The white dots show the same thing another way. They are 60 repeats of the study with a model whose true sensitivity is exactly 90%, and they scatter widely, because each repeat rests on 20 patients. One missed case more or less moves the result by 5 points.

Compare the grey bar. The same 90%, measured on the 180 patients without the condition, has an interval of 85% to 94%. The study is large enough for specificity and too small for sensitivity. Now move the number of patients to 1,000: there are 100 positive cases, and the interval is 83% to 94%, entirely above the requirement. Or return to 200 patients and raise the prevalence to 30%, as in a study enriched with known cases: 60 positive cases give 80% to 95%, with a lower end of 79.9% that is still just short of the requirement.

The correct practice is to plan from the positive cases. The demo reports that a ±5 point margin at 90% sensitivity needs about 141 positive cases, about 1,410 patients at 10% prevalence. Fix the minimum acceptable value before the study, report every estimate with its interval, and judge the result by the lower end of the interval. If cases are added to enrich the sample, say so, because predictive values then no longer reflect the real prevalence.

• How many positive cases does the validation set contain, and how many negatives?
• Is every sensitivity, specificity and AUC reported with a confidence interval?
• Was the minimum acceptable value fixed before the study, and is the lower end of the interval above it?
• Was the sample size planned from the expected prevalence and the precision needed?
• If the sample was enriched with cases, are predictive values recalculated for the real prevalence?
• Does each subgroup you report on hold enough cases for its own interval to mean something?

Related: our clinical model validation service, the wiki page on medical AI validation, and the demos on hidden subgroup failure and calibration and decision thresholds.

How many patients do I need to validate a clinical AI model?

It depends on how many of them have the condition and how precise the estimate has to be. In this demo, measuring a sensitivity of 90% to within ±5 points takes about 141 positive cases, which is about 1,410 patients at 10% prevalence and about 470 at 30%. Work out the positive cases first, then the patients needed to obtain them.

Why is the confidence interval for sensitivity so wide?

Because sensitivity is measured only on the patients with the condition, and there are usually few of them. In the demo 200 patients at 10% prevalence contain 20 positive cases, and 90% measured on 20 cases has a 95% interval of 70% to 97%. The same value on the 180 negatives has an interval of 85% to 94%.

Is a sensitivity of 90% on 20 positive cases enough to show a model works?

Not if the requirement is above about 70%. The 95% interval for 18 of 20 runs from 70% to 97%, so the data are consistent with a model that detects only seven cases in ten. A claim is supported when the whole interval lies above the minimum you set in advance.

What is a Wilson confidence interval?

It is a standard way to compute a confidence interval for a proportion such as sensitivity or specificity. Unlike the simplest formula, it stays inside 0% to 100% and behaves sensibly with small samples and with values close to 100%, which is the usual situation in validation studies. The demo uses the 95% Wilson interval throughout.

Does enriching the validation set with positive cases help?

It narrows the interval for sensitivity, because there are more positive cases to measure it on. In the demo 200 patients at 30% prevalence give 60 positive cases and an interval of 80% to 95%. The cost is that predictive values and the number of alerts no longer match real use, so they must be recalculated for the true prevalence.