Predictive value and prevalence: 3D demo

Sensitivity and specificity describe the test: how often it is positive in people with the condition and negative in people without it. Predictive values describe a result: the chance that a person who tests positive has the condition, and that a person who tests negative does not. The two are linked by the prevalence, the share of the tested population that has the condition.

The surface shows the positive predictive value for every combination of prevalence, from 0.1% to 50% on a log scale, and specificity, from 50% to 99%, at the sensitivity you set. It is low and flat across the whole rare-condition side and rises steeply only where the condition is common or the specificity is very high. The amber marker is the test and population you chose, and the readouts give the same result as counts per 1,000 people tested.

This matters whenever a model moves between populations. A study enriched with cases has a high prevalence and reports a high predictive value. In screening the prevalence can be a hundred times lower, and the same model then flags many people for each real case. A validation report should give predictive values at the prevalence of the intended use and say how many people are flagged per true case. See clinical model validation.

Picture a team with a model that detects a condition with 90% sensitivity and 90% specificity. In their validation study half the patients had the condition, and nine in ten positive results were correct. They plan to use the model to screen a general population in which about one person in a hundred has the condition.

The demo’s default settings are that screening population: 1% prevalence, 90% sensitivity and 90% specificity. Positive predictive value reads 8.3%. Of 1,000 people tested, 108 test positive and 9 of them have the condition. There are 11 false positives for every true one.

Nothing about the model has changed between the study and the screening programme. In 1,000 people there are 10 with the condition, and the model finds 9. There are 990 without it, and 10% of those, 99 people, test positive as well. The negative predictive value is 99.89%, so a negative result is very reliable, and 1 person with the condition is missed.

Move the prevalence to 10%, as in a referred group: positive predictive value is 50%. At 30% it is 79%. Return to 1% and raise the specificity to 99% instead: positive predictive value becomes 48% and positives per 1,000 falls to 19. On the surface these are the two ways up from the flat region: more disease in the tested group, or far fewer false positives.

The correct practice is to report predictive values at the prevalence of the population the model will be used in, not the prevalence of the study. State the number of positives per 1,000 people and how many of them are real, and plan what happens to the false positives, because each one is a person sent for further testing. For a rare condition, specificity matters more than a few points of sensitivity.

• What is the prevalence in the population where the model will be used, and how do you know?
• Are predictive values reported at that prevalence, not at the study’s prevalence?
• How many people per 1,000 will be flagged, and how many of them have the condition?
• What follow-up does each false positive receive, and what does it cost the patient and the service?
• Was the validation set enriched with cases, and are the figures corrected for it?
• Would a second, more specific test for positives change the balance?

Related: our clinical model validation service, the wiki page on medical AI validation, and the demos on calibration and decision thresholds and alert burden.

Why is the positive predictive value low when the disease is rare?

Because the people without the disease vastly outnumber those with it, so even a small false-positive rate produces more false positives than there are true positives. In this demo, at 1% prevalence a test with 90% sensitivity and 90% specificity gives 99 false positives and 9 true positives per 1,000 people, a positive predictive value of 8.3%.

How do I calculate positive predictive value from sensitivity, specificity and prevalence?

Multiply prevalence by sensitivity to get the true positives, and multiply (1 − prevalence) by (1 − specificity) to get the false positives. Positive predictive value is the true positives divided by the sum of the two. With 1% prevalence, 90% sensitivity and 90% specificity that is 0.009 divided by 0.108, or 8.3%.

If a test is 90% accurate, what is the chance a positive result is correct?

It depends on how common the condition is among the people tested. With 90% sensitivity and 90% specificity the demo gives 8.3% at 1% prevalence, 50% at 10% and 79% at 30%. The accuracy of the test alone does not answer the question.

Does sensitivity or specificity matter more for screening a rare condition?

For the positive predictive value, specificity matters far more. In the demo, at 1% prevalence, raising specificity from 90% to 99% lifts the positive predictive value from 8.3% to 48%. Sensitivity decides how many cases are missed, so both have to be stated, but false positives dominate the workload when the condition is rare.

Why do predictive values from a study not apply to my population?

Studies are often enriched with cases, so their prevalence is much higher than in routine use, and predictive values depend directly on prevalence. Sensitivity and specificity transfer more readily, although they can also change with case mix. Recalculate the predictive values with the prevalence you expect.