Diagnostic tests for rare conditions present a mathematical conundrum

Worried about your health? You can now avail yourself of noninvasive diagnostics that claim to screen for rare birth defects, cancer-associated mutations, and even Alzheimer’s. Even before the Covid pandemic, a 2018 paper reported that new genetic tests, many of them for increasingly rare conditions, were being released at the rate of 10 a day. These tests can be both sold direct to consumer and ordered by your physician.

A significant factor helping drive this entrepreneurial proliferation is that so-called lab-developed tests, which are designed, manufactured, and used in the same facility, have been exempt from FDA oversight. Consequently, such “LDTs” can be quickly brought to market without the FDA review of their effectiveness, labeling accuracy, and marketing claims required for regular tests.

In September, the FDA proposed to remove this loophole. In particular, the agency mentioned LDTs’ “false positive” prenatal diagnoses of rare genetic disorders that, instead of delivering the peace of mind promised, have thrown several expecting parents into a panic and even resulted in the abortion of healthy fetuses. Also cited were problems with LDTs in other contexts, such as inaccurate genetic testing that has led to ineffective or harmful cancer treatments. The proposed rule is open until Dec. 4 for public comments; after reviewing these, the FDA will decide whether to withdraw, revise, or finalize the rule.

While a great many LDTs are routinely used without incident by doctors and hospitals, stronger FDA oversight is long overdue. Certainly, it should help tamp down the exaggerated claims used to market some LDTs.

However, by considering the problem “solved” as long as LDTs and non-LDTs are treated similarly, the FDA proposal obscures a fundamental mathematical limitation to testing, which too many patients and doctors are unaware of: Practically all tests, not just LDTs, carry the risk of false positives, which can render the results effectively useless when the condition is rare enough. For such conditions, many tests should only be used when additional risk factors or symptoms are present.

Let’s explain why in terms of the home antigen test for Covid. Due to unavoidable variations in biological specimens and their laboratory processing, any test will give a certain percentage of false positive results. Let’s say this rate is 1% for our antigen test (which is within the range seen in practice). When the incidence of Covid is high enough, this 1% is not a major problem, in the sense that if you test positive, you probably have Covid. However, as the incidence decreases, the false positives begin to dominate, since there are fewer true positives around. As an extreme case, imagine if Covid had been entirely eliminated (we wish). Then, 1% of people who decided, on a lark, to deploy their unexpired leftover tests would still get a positive result. None of them would actually have Covid, which means that if you got a positive result, there would be a 0% chance of it being correct.

For rare genetic conditions, in the absence of additional risk factors, the situation is similar. For instance, suppose you used one of several prenatal LDTs marketed to routinely check for such genetic birth defects like Prader-Willi or Wolf-Hirschhorn syndromes, each of which has an incidence of only about 1 in 20,000 births (i.e. 0.005%). Let’s say this test had a similar false positive rate of 1% (again comparable to what may be observed in practice). If the test came back positive, then the chance of it being a true positive would again be almost zero. More precisely, 199 out of 200 positive results, on average, would be wrong.

This means that for most people the test is essentially worthless, and certainly no medical decision (like a pregnancy termination) should be made based only on it. Rather, a positive test needs to be followed up by more decisive tests, which, if they exist, are generally both expensive and invasive, and may even carry a small risk of miscarriage. This follow-up doesn’t always happen.

A probable reason for this is that many doctors misinterpret a 1% false positive rate to mean that a positive result has a 99% chance of being correct. As research from one of us, Daniel, and his colleagues has shown, such lack of statistical understanding leads even physicians to vastly overestimate the risk of conditions like heart attacks, breast cancer, and infections. As a result, they often prescribe unnecessary antibiotics and other treatments. Patients also make the same errors in estimations.

Further confusing the issue is that prenatal LDTs have a successful track record of testing for some other birth defects like Down syndrome. But there is a reason for this: Down syndrome is almost 30 times as frequent as the rare conditions discussed above.

Let us suggest three remedies. The first is for the FDA to require all tests (to the extent possible) to carry clear-cut, easily comprehensible labeling on what a positive result really means — i.e., what percentage of these can be expected to be true positives. A similar disclosure for negative results should also be mandatory; false negatives can be a significant problem for less rare conditions.

The second is to improve statistical understanding of medical issues in both physicians and patients. To this end, one of us, Daniel, has designed an array of interactive teaching tools on how to estimate the effects of false positives and negatives — these have been particularly successful with doctors in training. Manil has found that college students will engage with this topic in earnest as part of a math course, even when they are conspicuously disinterested in math generally. Perhaps they realize that plotting a quadratic isn’t going to save their lives, but this might.

The third is to repudiate the idea of testing being a panacea. Too many patients think they should go in for as many tests as their insurance will cover, in order to guard against even outlandishly improbable conditions. People need to understand that testing is a gamble, and they need to know how to make a smart bet.

Eighteenth-century mathematician Daniel Bernoulli was the first to realize the connection between medicine and games of chance (though awareness of the uncertainty inherent in medicine has existed since Hippocrates). Patients are often exposed to the concept of treatments being a gamble: If a cancer surgery has a 1 in 10 chance of saving their life, then they usually can understand that they may be one of the other nine, should they decide to take the risk.

Such thinking in terms of gambling and risk needs to become commonplace for diagnostic testing as well. Recall the genetic abnormality with 0.005% prevalence — if you don’t have additional risk factors, then without testing you can be 99.995% sure your baby won’t have it. People should be taught to be satisfied with the reassurance this percentage brings. Trying to increase it to 100% by adding to the 14 billion tests annually performed in the U.S. risks the agonizing consequences of testing positive for a disease that most likely isn’t there. Trust the probabilities more than you trust the test, in other words.

It is impossible to come by 100% assurance in much of medicine, so there’s no avoiding risks in diagnosis. But you can improve your outcomes by knowing which bets are not worth placing.

Manil Suri is a mathematics professor at the University of Maryland, Baltimore County, and the author of “The Big Bang of Numbers: How to Build the Universe Using Only Math.” Daniel Morgan is a physician; a professor of epidemiology, public health, and infectious diseases; and director of the Center for Innovation in Diagnosis, at the University of Maryland School of Medicine.