Statistics and CoV antibody testing

[ex-khobar Andy]

We are seeing a number of studies indicating that there are far more people around with antibodies than can be accounted for by known infection rates. There was one in Iceland a while ago and recently one in Santa Clara (CA) county. Before we all go off into 'herd immunity' wonderland, here's a pinch of salt.

Suppose that you have a test for antibodies. It's a very good test: it's 95% accurate. (That's typical for tests of this type. HIV-positive by immunoassay is presumptive and it is confirmed by a second test. This used to be Western Blot but is now more usually RNA-based. And we have 30 years experience looking for HIV.)

For very good public health reasons, medical tests are biased to give false positives rather than false negatives. It's generally thought better to treat non-cases than to let a carrier out into the neighborhood who might unknowingly spread the disease.

So the test is knowingly skewed towards the false positive side. So if it's 95% maybe it's 99.7% accurate for positives and 95.3% for negatives.

Suppose you have a population which is 0.2% affected with the virus. We have to date a US known cases number of something like 700,000 in a population of 350 million, so that's about right. But if you use this test to look for the antibodies on 1000 people, it will correctly find it in those 2 people (0.2%) who have had it or who currently have it. But it will also find a further 47 people (100% - 95.3% = 4.7%) who are false positives - who do not have it but appear to do so, based on the test.

The immediate conclusion, particularly from those who (a) do not understand statistics and (b) want some good news (and let's call this subgroup of the population politicians, for want of a better word) is that there are something like 25 times the expected number of people out there bearing the antibodies and that we are well on our way to herd immunity.

Not necessarily.

Edited to add: I went looking on the web to see if anyone was rising the same concern about the Santa Clara study and found this. (Google coronavirus statistics Bayesian - that's the abbreviated name of this statistical phenomenon) and found this piece by someone called Andrew who is (a) not me and (b) actually knowledgeable about statistics.

[ex-khobar Andy]

And this piece in the Los Angeles Daily News.

Unfortunately some people will dismiss this as a nerd-fest but these are real statistical issues with the potential for massive mistakes in public policy.

[Darren]

Article dated April 17th.

"Alex Greninger, the Virology Lab’s assistant director, said his team has run about 400 blood specimens through Abbott’s instruments, including samples that were stored from pre-COVID-19 blood tests. None of those old blood samples came back positive, but the test correctly identified people who were known to have had the virus.

Abbott’s internal study, involving 1,200 specimens, had a sensitivity of 100% to COVID-19 antibodies, Greninger said. Just as importantly, the test achieved a 99.6% specificity, meaning that it was almost always able to distinguish between SARS-CoV-2 and other viruses.

Once a blood specimen is drawn and delivered to the lab, it takes about 10 to 15 minutes to spin the sample to produce the serum for testing, and another 20 to 25 minutes to conduct the test, Greninger said. “The hardest part here is going to be getting the blood,” he said.

One issue that’s come up with regard to virus testing has to do with the availability of supplies, such as nasal swabs and reagents, but Jerome didn’t think this would be a problem for Abbott’s antibody test. “We’ve been assured that this pipeline is robust,” he said. Abbott says it’s shipping out almost 1 million of the tests to U.S. customers this week, and will ramp up to a total of 4 million tests in April. It plans to ship 20 million tests per month by June."

https://www.geekwire.com/2020/univ-wash ... ntibodies/

[Darren]

"“We went from zero tests a day to … something. I don’t know, but well over 100,000 tests a day. In that one month,” said Dr. Alex Greninger, assistant director of the University of Washington’s Clinical Virology Laboratory."

...

"The tests for COVID-19 fall into two basic categories: direct tests, which look for the presence of the virus in a patient, and indirect tests, which look for antibodies that show a patient’s immune system has encountered the virus at some point. We’ll need both to safely emerge from the confines of social isolation, said Scott Becker, chief executive officer of the Association of Public Health Laboratories. The direct tests are critical for finding infected people, tracing who they have been in contact with and isolating them before they can pass the virus to others. The indirect (or “serological”) tests, administered across broad swaths of the population, can help public health experts understand how the virus spreads and how people build immunity to it.

https://fivethirtyeight.com/features/th ... 7M9_VptTkQ

[ex-khobar Andy]

We are seeing a number of studies indicating that there are far more people around with antibodies than can be accounted for by known infection rates. There was one in Iceland a while ago and recently one in Santa Clara (CA) county. Before we all go off into 'herd immunity' wonderland, here's a pinch of salt.

Suppose that you have a test for antibodies. It's a very good test: it's 95% accurate. (That's typical for tests of this type. HIV-positive by immunoassay is presumptive and it is confirmed by a second test. This used to be Western Blot but is now more usually RNA-based. And we have 30 years experience looking for HIV.)

For very good public health reasons, medical tests are biased to give false positives rather than false negatives. It's generally thought better to treat non-cases than to let a carrier out into the neighborhood who might unknowingly spread the disease.

So the test is knowingly skewed towards the false positive side. So if it's 95% maybe it's 99.7% accurate for positives and 95.3% for negatives.

Suppose you have a population which is 0.2% affected with the virus. We have to date a US known cases number of something like 700,000 in a population of 350 million, so that's about right. But if you use this test to look for the antibodies on 1000 people, it will correctly find it in those 2 people (0.2%) who have had it or who currently have it. But it will also find a further 47 people (100% - 95.3% = 4.7%) who are false positives - who do not have it but appear to do so, based on the test.

The immediate conclusion, particularly from those who (a) do not understand statistics and (b) want some good news (and let's call this subgroup of the population politicians, for want of a better word) is that there are something like 25 times the expected number of people out there bearing the antibodies and that we are well on our way to herd immunity.

Not necessarily.

From today's CNN: CDC has finally glommed onto what I was describing a month ago.

The CDC explains why testing can be wrong so often. A lot has to do with how common the virus is in the population being tested. "For example, in a population where the prevalence is 5%, a test with 90% sensitivity and 95% specificity will yield a positive predictive value of 49%. In other words, less than half of those testing positive will truly have antibodies," the CDC said.