Innumeracy

I read Innumeracy many years ago. One of the parts that really stood out to me was the discussion of how even very accurate tests for diseases do not do as good a job as we generally think they do at telling who has the disease. I was reminded of this while reading a Robin Hanson blog this morning about how people have trouble working problems in probability. The example problem was for a test that was 80% true-positive rate and a 9.6% false-positive rate (correctly identifies 80% of the people with the disease, incorrectly reports the disease in 9.6% of people who do not have it) in a population where 1% have the disease. In this example it turns out that only about 8% of people the test reports as having the disease will actually have it: 10 out of 1000 have it, 8 of those test positive; 990 of 1000 do not have it, 95 of those test positive. So 8 of 103 that test positive actually have the disease.

In this case the example disease was breast cancer and the test was a mammogram. I wondered whether the numbers used were actual rates. I found several references that indicate that the rates in the example are very close to the actual rates except for the frequency in the population which varies quite a lot by age. The frequency in the example is about right for women over 40. The frequency for women under 40 seems to be about 1 in 1000 (0.1%). That changes the results somewhat making the test's predictions much worse: 10 out of 10000 have it, 8 of those test positive; 9990 of 10000 do not have it, 959 of those test positive. So 8 of 957 that test positive actually have the disease. By age 50 the incidence in the population jumps to 1 in 54 (about 2%) so: 20 of 1000 have it, 16 of those test positive; 980 don't have it, 94 of these test positive; 16 out of 110 get a positive test. Even at age 50 the predictive power of the test is pretty weak. Above 50 the incidence rates increase dramatically and so the predictive power of the test improves.

It is also interesting that physical exams are generally better that mammograms with a 90% true-positive rate and a 4% false positive rate. 10 out of 10000 have it, 9 of those test positive; 9990 of 10000 do not have it, 399 of those test positive. So 9 of 407 that test positive by physical exam actually have the disease. That's about a 50% reduction in errors, but still not a really good rate.

So if either of these tests say you've got it and you are 50 or younger, you should not be very worried until follow up tests (more invasive, but more accurate with fewer false positives) say you should be worried. This does not speak to whether it is worth taking the test. It almost certainly is since they are relatively cheap (especially if you have decent medical benefits) and the cost of not taking the test could be very very large (much more expensive treatments or death).

You can also flip the question around and ask if someone show feel really safe if she takes the test is told she does not have the disease. 2 of the 10 with the disease will have a false-negative, 895 of the rest will be correctly told they are clean, so 2 of the 897 given a clean slate are really not. That's not bad. If the test says you are clean you are pretty safe not worrying about it.

And remember: I am not a doctor. I don't even play one on television. I can't hold a candle to one that plays poker or even to a non-doctor poker doctor.

I found these interesting articles while researching this:

National Cancer Institute - see question number 10.
False-Positive Mammogram Results Vary Among Radiologists
The Role of Numeracy in Understanding the Benefit of Screening Mammography
Evidence Based Medicine See Module #2, section #4
Breast Cancer: Statistics on Incidence, Survival, and Screening