Alert burden: interactive demo

An early-warning model scores each patient repeatedly and raises an alert when the score crosses a threshold. The threshold is usually chosen to reach a target sensitivity, and the report gives the AUC and that sensitivity. Neither number says how often the alert sounds. That depends on three more things: how rare real events are, how many patients a clinician looks after and how often each one is scored.

In the demo a real event happens in 2% of patient-shifts. With six patients scored every hour, a nurse’s shift contains 72 scores, almost all of them for patients who are fine. Even a small false-positive rate on 72 scores gives many alerts, while real events arrive about once every nine shifts. The chart shows one simulated week for one nurse, with each alert as a mark.

When nearly every alert is false, alerts stop being acted on, which is the problem usually called alarm fatigue. The remedies are a better model, a higher threshold with a stated cost in missed events, less frequent scoring, or suppressing repeats for the same patient. Each is a decision for the clinical team, and it can only be made when the threshold is expressed as workload. See clinical model validation.

Picture a hospital introducing a model that warns of patient deterioration. The validation report shows an AUC of 0.90. The clinical lead asks for a threshold that catches 90% of events, which sounds like a careful choice, and the model is switched on for a ward where each nurse looks after six patients.

The demo’s default settings are that ward: AUC 0.90, 90% sensitivity, 6 patients per nurse, each scored every hour. Alerts per nurse per shift reads 22, and Time between alerts reads 33 min. Real alerts reads 1 in 199. Real events missed reads 10%, which is what a sensitivity of 90% means.

The chart shows a simulated week for one nurse: 151 alerts, of which 0 are real. That is not bad luck. With a real event in 2% of patient-shifts, a nurse with six patients meets one about once every 9 shifts, and in between every alert is false.

Now look at what would change it. A model with an AUC of 0.95 gives 11 alerts a shift, 1 in 100 real. At 0.98 it is 3.9 a shift, 1 in 36. Keeping the 0.90 model and lowering the sensitivity to 80% gives 12 alerts a shift, 1 in 125 real, at the price of missing 20% of events. Scoring once a shift instead of hourly gives 1.9 alerts, 1 in 17 real.

The correct practice is to choose the threshold with the people who will receive the alerts. Express it as alerts per clinician per shift and real events per alert for the actual staffing and scoring frequency. Agree what load can be acted on and what miss rate is acceptable. Consider suppressing repeat alerts for a patient who has already been reviewed, and measure the alert rate and the response to alerts after go-live.

• How many alerts per clinician per shift does the chosen threshold produce on your ward?
• How many alerts are raised for each real event?
• How often is each patient scored, and are repeat alerts for the same patient suppressed?
• What share of real events is missed at this threshold, and is that accepted by the clinical team?
• What is a clinician expected to do for each alert, and how long does it take?
• Are the alert rate and the response to alerts measured after deployment?

Related: our clinical model validation service, the wiki page on medical AI validation, and the demos on calibration and decision thresholds and predictive value and prevalence.

How many alerts will a clinical prediction model generate?

Multiply the number of scores per clinician per shift by the share that exceed the threshold. That share depends on the model’s AUC, the sensitivity you ask for and how rare events are. In this demo an AUC of 0.90 at 90% sensitivity, with 6 patients scored hourly, gives about 22 alerts per nurse per 12-hour shift.

Why does a model with a high AUC cause alarm fatigue?

Because AUC says nothing about how rare the event is or how often patients are scored. When events are rare, even a modest false-positive rate applied to many scores produces far more false alerts than real ones. In the demo about 1 alert in 199 is for a real event.

How do I choose a threshold for an early-warning score?

Start from the workload the ward can act on and the miss rate it can accept, and find the threshold that fits both. In the demo, lowering sensitivity from 90% to 80% cuts alerts from 22 to 12 a shift and raises missed events from 10% to 20%. That trade-off is a clinical decision and should be made explicitly.

How can alert burden be reduced without missing more events?

Improve discrimination, score less often, or suppress repeat alerts for a patient who has already been reviewed. In the demo, raising the AUC from 0.90 to 0.95 at the same 90% sensitivity halves the alerts, from 22 to 11 a shift. Scoring once a shift instead of hourly cuts them to 1.9, although a real system would then detect events later.

What is a reasonable number of alerts per nurse per shift?

There is no universal figure; it depends on what each alert requires and on everything else the nurse is doing. What matters is that the number is known before deployment and agreed with the ward. The demo reports it alongside the time between alerts and the number of alerts per real event.