Hidden subgroup failure: interactive demo

The 6,000 synthetic patients belong to a main group or to a subgroup, which stands for any group the training data covered poorly: an age band, a site, a device, a disease stage. The model’s score separates cases from non-cases well in the main group and by only a share of that in the subgroup, set by the signal slider. Prevalence is the same in both groups, so the difference comes from the model and not from the case mix.

One threshold is used for everyone. It is set where the model reaches the target sensitivity on the whole test set, which is how a threshold is usually chosen. Because the subgroup’s cases get lower scores, more of them fall below that threshold, and the main group’s sensitivity rises above the target to make up the average. The overall figure is correct and describes neither group.

The smaller the subgroup, the less it moves the overall score and the fewer cases it holds, so the failure is both better hidden and harder to measure. The demo reports the subgroup’s sensitivity with a 95% interval for that reason. Good practice is to name the subgroups before looking at the results, report discrimination and error rates for each one, and check that each holds enough cases to support a conclusion. See clinical model validation.

Picture a team validating a risk model on a large test set. The patients come from several age bands, sites and devices, in the proportions seen in practice. The report gives one AUC and one sensitivity for the whole set, both good, and the model is approved.

The demo’s default settings show what that report can hide. Of the 6,000 synthetic patients, 922 belong to a subgroup in which the model’s signal is 30% of what it is elsewhere. Overall AUC reads 0.87 and overall sensitivity 90%, which is the target the threshold was set to reach. Subgroup AUC reads 0.66 and subgroup sensitivity 59%.

The line under the readouts gives the counts. In the subgroup the model misses 83 of 204 cases; in the main group it misses 36 of 988. The bars show how the overall figure is made: sensitivity is 96% in the main group and 59% in the subgroup, and the average of the two, weighted by size, is the 90% in the report. Specificity is 62% and 66%, so the subgroup is not being flagged more often either.

Now shrink the subgroup to 5% of patients. Overall AUC rises to 0.91, because the weak group counts for less, while subgroup sensitivity falls to 51%: 34 of 69 cases are missed. The interval for that figure is 39% to 62%, wide because there are only 69 cases. The smaller the group, the better the overall score looks and the less can be said about the group. Set the signal to 100% and the difference disappears: subgroup AUC is 0.92 and sensitivity 91%.

The correct practice is to decide the subgroups before looking at the results, from what could plausibly change the model’s behaviour: age, sex, site, device, disease stage, care setting. Report discrimination, sensitivity and specificity for each one with its interval, at the threshold that will be used. Where a subgroup is too small to judge, say so and collect more cases, and do not treat an overall pass as covering it.

• Were the subgroups named before the results were seen, and do they cover age, sex, site and device?
• Are AUC, sensitivity and specificity reported for each subgroup, not only overall?
• Is each subgroup figure given with a confidence interval and the number of cases behind it?
• Is the threshold the one that will be used in practice, applied to every subgroup?
• Does the test set contain enough cases from each subgroup the model will meet?
• Is subgroup performance monitored after deployment, as the patient mix changes?

Related: our clinical model validation service, the wiki page on medical AI validation, and the demos on sample size for validation and patient-level data leakage.

Why does my model perform well overall but fail on a subgroup?

Because the overall figure is an average weighted by group size, and a small group has little weight. In this demo a subgroup of 15% of patients has an AUC of 0.66 and a sensitivity of 59%, while the overall figures are 0.87 and 90%. Common causes are too few examples of the group in training, different measurement devices or sites, and a different presentation of the disease.

How do I check a clinical AI model for subgroup performance?

Name the subgroups in advance, then compute discrimination and error rates for each one at the threshold you intend to use, with confidence intervals. In the demo the per-group view shows 96% sensitivity in the main group against 59% in the subgroup, a difference the overall 90% does not reveal. Check that each group holds enough cases for its interval to be useful.

Can a high overall AUC hide poor performance in a minority group?

Yes, and the smaller the group, the better it is hidden. In the demo, shrinking the subgroup from 15% to 5% of patients raises the overall AUC from 0.87 to 0.91 while the subgroup’s sensitivity falls from 59% to 51%.

Which subgroups should a medical AI validation report?

The ones that could plausibly change how the model behaves and that matter for its intended use: typically age bands, sex, sites, scanner or device types, disease stage or severity, and care setting. Choose them before analysing the results, so the report is not limited to the groups where the model happens to do well.

How many patients does a subgroup analysis need?

Enough cases in each subgroup for the interval around its result to be informative. In the demo a subgroup with 204 cases gives a sensitivity interval of 52% to 66%, and one with 69 cases gives 39% to 62%. If a subgroup is too small to judge, the honest report says so.