Shortcut learning: interactive demo

Each synthetic image carries two things a model could use. One is a disease signal, which is real but weak: on its own it supports an AUC of 0.77. The other is a tag in the corner of the image that says nothing about the patient but, in the training hospitals, appears mostly on diseased cases. That happens in real data when sicker patients are imaged on a portable machine, at a different site or with different markers, tubes or burned-in text.

The model is free to use both, and the tag is the easier one. With the tag going with the diagnosis 90% of the time, most of the model’s weight lands on it. Tested on new patients from the same hospitals, the model looks better than the disease signal alone could make it, because the shortcut still works there. At a new hospital the tag appears on healthy and diseased patients alike, and the model’s reliance on it now adds errors.

An internal test set cannot reveal this, because it shares the shortcut. What reveals it is a test where the link is broken: another site, another machine, or the same images with the tag removed. Checking what the model responds to, and controlling how cases and controls are acquired, are the other defences. See medical imaging AI.

Picture a team that collects chest images from two hospitals to train a detector. Most of the diseased cases come from an intensive-care unit, where patients are imaged in bed on a portable machine that stamps a tag in the corner. Most healthy cases come from an outpatient clinic with a fixed machine and no tag. The team splits the images at random into training and test sets.

The demo’s default settings reproduce that: the tag goes with the diagnosis 90% of the time and the disease signal is 1.0. AUC, same hospitals reads 0.94. That is better than the disease signal can support on its own: AUC without the tag reads 0.77. The extra performance comes from the tag, and Weight on the tag reads 70%.

Now the model is installed in a hospital where every patient is imaged on the same machine, so the tag has nothing to do with disease. AUC, new hospital reads 0.64. That is not only lower than the 0.94 from the internal test. It is lower than the 0.77 of a model that never saw the tag, because the model keeps reacting to a tag that now misleads it.

Move the first slider to 50%, where the tag is unrelated to the diagnosis in training as well. Weight on the tag falls to 3% and the two AUCs are 0.76 and 0.77. Move it to 98% and the same-hospital AUC is 0.99 with 82% of the weight on the tag, while the new-hospital AUC stays at 0.64. The more convenient the shortcut, the better the internal result looks and the worse the model is.

The correct practice is to look for shortcuts before trusting a result. Compare how cases and controls were acquired: site, machine, view, markers, text and time period. Test on data where those links are broken, ideally from another site. Check what the model responds to, for example by masking the corners or the text and scoring again. Where a shortcut is found, remove it from the images or balance the acquisition, then validate again.

• Do cases and controls come from the same sites, machines, views and time periods?
• Do the images contain tags, text, markers, tubes or devices that go with the diagnosis?
• Was the model tested on a site where those links do not hold?
• Does performance survive masking the image borders, corners and burned-in text?
• Do attribution maps point at anatomy, or at the edges of the image?
• Is the internal result better than is plausible for the task?

Related: our medical imaging AI service, the wiki page on medical AI validation, and the demos on scanner shift and patient-level data leakage.

What is shortcut learning in medical AI?

It is when a model predicts the label from a feature that accompanies the diagnosis in the training data without being part of the disease, such as a hospital tag, a type of machine or a chest drain. The model scores well wherever that link holds and fails where it does not. In this demo the AUC is 0.94 at the training hospitals and 0.64 at a new one.

Why does my medical imaging model work in testing but fail at another hospital?

One common reason is that the test set came from the same hospitals as the training set and shares their shortcuts. If sicker patients were imaged differently, the model may have learned the difference in acquisition. In the demo a tag that goes with the diagnosis 90% of the time attracts 70% of the model’s weight, and the AUC falls by 0.30 at a hospital where the tag is uninformative.

How do I detect that a model is using a spurious feature?

Test it where the feature no longer goes with the diagnosis: another site, or the same images with the feature masked. Compare the result with the internal test. Attribution maps that highlight corners, borders or text instead of anatomy are a further sign, as is an internal result that looks too good for the task.

Can a shortcut make a model worse than a model without it?

Yes. In the demo the model that learned the tag scores 0.64 at a new hospital, while a model trained without the tag scores 0.77 there. The shortcut does not just stop helping: the weight placed on it turns into errors when the tag appears on healthy patients.

How do I prevent shortcut learning?

Control the data before training: acquire cases and controls in the same way where possible, or balance and match them by site and machine. Remove burned-in text, tags and borders from the images. Then validate externally. No training method guarantees that a model ignores a shortcut that is present in the data.