The Indian Council of Medical Research (ICMR) recently made a welcome announcement: that it had built a testing kit to detect antibodies against the novel coronavirus in human blood samples. The kit, the agency said in its press release, was 100% specific and 98% sensitive.
A 100% specificity means that the test will never confuse other antibodies for those specific to the novel coronavirus. A sensitivity of 98% means when the kit checks a hundred blood samples containing the antibodies, it will detect the antibodies 98% of the time.
These are impressive claims. Few antibody tests anywhere in the world today boast of such accuracy. For example, the Elecsys antibody kit built by Roche, a global leader in medical diagnostics, claims 99.8% specificity and 100% sensitivity. ICMR’s kit seems to trump Roche’s on the first count. Such high accuracy should mean that the agency’s kit is perfect to estimate the true prevalence of COVID-19 in the country – a crucial question at this juncture. Because so many people with COVID-19 show no symptoms, it’s very hard to say how many Indians have been infected thus far.
An accurate antibody kit can solve this problem because the presence of antibodies specific to the novel coronavirus in a person’s blood would indicate that they’ve been exposed to the virus in the past.
ICMR plus various state governments have been planning surveys to estimate the percentage of Indians exposed to COVID-19 for a while now. These surveys are known as seroprevalence, or ‘sero’, surveys. However, after the low accuracy of Chinese antibody kits that ICMR had imported came to light, the idea was shelved.
A kit that is 100% specific allows ICMR to get back on track with its sero surveys. Indeed, it’s close to completing one involving 24,000 people in 69 districts, Manoj Murhekar, the head of ICMR’s National Institute of Epidemiology in Chennai, told The Wire Science.
But how reliable is ICMR’s claim that its kit is 100% specific? Does it really mean the kit will correctly classify everyone who has never been exposed to the virus as negative? Unfortunately, the answer is complicated.
To verify this claim, we need to know how ICMR arrived at these numbers. And neither ICMR nor Zydus Cadila, the company contracted to manufacture the agency’s kits, has published these details.
And in the absence of this data, experts say, it’s hard to take the numbers at face value. “The devil is in the details,” Binay Panda, a genome scientist at the Bengaluru-based Ganit Labs, said.
How many samples?
The first thing we need to understand ICMR’s claim is: how many samples were used to calculate the kit’s specificity and sensitivity? (See box: What are sensitivity and specificity?). A researcher usually calculates specificity by using human blood samples that don’t contain antibodies specific to the novel coronavirus. One way to do this is to find blood samples collected from people before November 2019 – i.e. before the outbreak began.
“Finding negative samples is usually not a problem. Because we really believe this is a novel virus, anybody whose serum sample[footnote]The part of the blood not involved in clotting[/footnote] is available from before November can be assumed to be negative,” Jacob John, a professor of community medicine at the Christian Medical College (CMC), Vellore, said.
The researcher then runs these samples through the antibody test and counts the number of times the test classifies the samples correctly.
To calculate sensitivity, the researcher does the opposite: she turns to people she knows have antibodies against the novel coronavirus. This is harder because no perfect or ‘gold standard’ antibody test exists today that can tell for sure if an individual has mounted an immune response. So most kit-manufacturers use the next best thing: the reverse transcriptase polymerase chain reaction (RT-PCR), which detects the virus’s genetic material in swabs from people’s noses and throats.
“We need to understand that RT-PCR and antibody kits are different tests that measure different aspects of the infection,” Devasena Ananthraman, a cancer researcher and epidemiologist at the Rajiv Gandhi Centre for Biotechnology, Thiruvananthapuram, said. “One detects whether the virus itself is present while the antibody test measures the host’s response to the virus.”
Luckily, there’s a way to predict when antibodies will show up after the virus has been detected in one’s body. Most COVID-19 research indicates that infected people develop antibodies a few weeks after exposure. So it’s reasonable to assume blood samples taken from them at this time will contain the antibodies.
The researcher tests samples like this – and by counting the number of samples the kit could not find antibodies in, the researcher estimates the kit’s sensitivity.
Now, the challenge with such an estimate is that it ought to be based on a large number of samples to determine how reliable the kit will be in the real world. This is because all diagnostic tests are imprecise, and their results vary from one person to another, and from one day to the next.
For example, the ICMR’s kit measures a type of antibody called Immunoglobulin G (IgG). Each infected person may have different IgG levels even if their IgG levels are measured on the same day after infection. So a kit that yields a positive result for one infected person may not do so for another.
To account for such variations, a diagnostic test must ideally be tested with multiple groups of people. And when this is not possible, researchers calculate the range across which the sensitivity and specificity values can vary for each kit, in addition to single estimates like 100% or 98%. This range is called the confidence interval.
Specificity is a measure of a test kit’s “true-negative” rate. A 100% specificity means that if 100 blood samples that lack antibodies to SARS-CoV-2 are tested with the kit, the kit will return a “negative” result all hundred times. This feature is important because antibody kits rely on an “antigen” – a protein on the surface of the SARS-CoV-2 virus– to detect the antibodies. When antibodies in a blood sample bind with the antigen, the kit gives a positive signal.
However, many other viruses, including the Middle Eastern Respiratory Syndrome (MERS) virus or the human coronavirus 229 E, could have similar antigens. So, the SARS-CoV-2 kit antigen must be unique enough that it doesn’t confuse antibodies to other viruses with the ones it is seeking. If the test does mix-up antibodies – or cross-reacts with them – it will give an incorrect positive result. Scientists call this a “false positive”.
Sensitivity, on the other hand, is a kit’s “true positive” rate. So, a 100% sensitivity means that if the test kit is run on 100 samples that have SARS-CoV-2 antibodies, the kit detects the antibodies in all of them. This feature is crucial too, because some kits don’t catch low levels of antibodies in blood samples, leading to what scientists call “false negatives”.
The question then becomes: what are the confidence intervals for the ICMR kit’s specificity and sensitivity? The agency never revealed this in its press release. However, in a phone interview, ICMR virologists Gajanan N. Sapkal and Pragya Yadav told The Wire Science that the sensitivity and specificity mentioned in the announcement had been calculated using 75 positive and 75 negative samples. That is, the ICMR’s antibody test rightly classified 75 out of 75 negative samples and 73-74 positive samples.
If these numbers are correct, the ICMR kit’s sensitivity ranges from 92.3% to 99% and specificity, from 95% to 100%, according to a statistical method called the exact Clopper-Pearson’s confidence interval. An alternative method, called the rule of three – used when a test returns no false positives – yields a confidence interval of 96%-100% for specificity, according to Gautam Menon, a computational biologist at Ashoka University, Sonipat.
In plain English, these confidence intervals mean that if the ICMR’s kit was used to test a different group of people, its sensitivity could drop to 92.3% and its specificity, to 95-96%. And this is a problem.
How low specificity can mess up estimates
In large parts of India today, COVID-19 is likely to have infected less than 5% of the population, say experts. In these places, if a test has a sensitivity of 93% and a specificity of 95% – both considered relatively low – most of its positive results are likely to be false positives.
One way to estimate how many false positives a test will throw up is the positive predictive value, which indicates how likely it is that an individual who’s positive has really been infected in the past. A prevalence of 5%, a specificity of 95% and a sensitivity of 93% yields a positive predictive value of 50%. This means for an individual getting a positive result, there’s a solid 50-50 chance that the result is incorrect.
If the prevalence of infected persons in a group is 1%, the positive predictive value drops to 16%, which means up to 84% of positive results would be incorrect. Ergo, if the ICMR kit is used in a place with 200,000 people, 1% of whom are actually infected, the actual number of people with antibodies would be 2,000 but the kits would add another 9,900 to this number.
This kind of scenario is not unlikely, according to John of CMC Vellore. While hotspots such as Gujarat’s Surat may well have a prevalence of more than 5%, there will be areas with prevalence even lower than 1%. “At such low prevalence, serology [antibody tests] will tell you nothing.”
India isn’t the only country where low local prevalence of COVID-19 is messing with sero surveys. A survey in Santa Clara county, California, by Stanford University researchers drew heavy flak when it published its initial results. The researchers had tested 3,330 participants in the county for antibodies; 50 tested positive. Based on this, they estimated the true number of infected people in the county to be between 50,000 and 85,000, compared to the 1,000 confirmed cases confirmed with RT-PCR tests.
But other scientists quickly pointed out that the test’s specificity in the Santa Clara survey had a wide confidence interval. Per one calculation, the lower bound of the confidence interval could be lower than 98.8%, which could mean all 50 people with antibodies in that survey could have been false positives.
Stanford researchers then revised their paper by including more data on the test they had used, which was manufactured by Hangzhou Biotest Biotech and distributed by a US firm named Premier Biotech. This new data increased the sample size for the kit’s specificity and raised the lower bound of the confidence interval to 99.2%.
Concerns about test specificity have also put paid to sero surveys being planned in India. For instance, Kerala was preparing to survey 200,000 people using one of the rapid antibody-testing kits ICMR had imported from China. This kit eventually turned out to be highly inaccurate, prompting ICMR to recall them from around India.
ICMR’s current kit uses a technology called the enzyme-linked immunosorbent assay (ELISA), thought to be more reliable than the technologies in most rapid antibody-testing kits. But it doesn’t make the ICMR test infallible either.
Should ICMR have validated its kit better?
Given how tricky sero surveys can be with a fallible kit, ICMR ought to have tested its kit with more samples, experts told The Wire Science. “A sample of 75 people [for specificity] isn’t good enough,” John said.
Ravikumar Banda, the owner of Bangalore-based XCyton Diagnostics, which is also developing an ELISA kit, said he was surprised ICMR chose to validate its kit with only 75 samples given the agency’s network of labs used to perform most of the COVID-19 tests in India until recently. “They held all the samples in India initially. They could have done their testing on a thousand samples,” he told The Wire Science.
Indeed, what is the ideal sample size? John calculated that if 370 negative samples were tested, and all were identified correctly, one could be relatively sure that the specificity wouldn’t drop below 99%. According to him, it is only at this level that false positives wouldn’t crowd out true positives in areas with low prevalence.
While different ELISA kits available in the market today have been validated with different sample sizes, some companies have tested their kits on quite a few. For example, Roche’s Elecsys kit was validated with 5,272 samples, landing on a specificity of 99.8% and confidence interval of 99.65-99.91%. However, the company validated the kit’s sensitivity with only 29 samples, so the confidence interval for this metric is a wide 88-100%.
Scientists also have other questions about ICMR’s kit. For one: what were the ages and symptoms of the COVID-19 patients from whom ICMR collected its test samples?
In April this year, Chinese scientists reported in a preprint paper that people older than 40 years had far more neutralising antibodies[footnote]Antibodies that can deactivate the virus[/footnote] in their blood compared to younger people. Other studies suggest there may be differences in antibody-response among people with mild and severe infections.
In effect, we don’t know if ICMR’s kit can perform equally well among all age-groups and across the spectrum of symptoms. This matters because the agency’s countrywide sero survey will encompass a wider set of demographic attributes.
Also read: No COVID-19 Test Is 100% Right, so Their Errors and Results Are Both Important
Then there’s the question of which other viruses were tested for cross-reactivity with the ICMR kit (see box for more). ICMR’s Sapkal told The Wire Science that some of the validation samples were from people who had previously been infected with respiratory viruses, including influenza and adenoviruses. While this is good news, the body hasn’t published any list of all viruses for which cross-reactivity was determined.
Srinivasan Govindaraman, a radiologist and the director of Anderson Diagnostics, Chennai, also said the sources of these samples matter. Those from healthy donors to a blood bank, for example, would have different characteristics from those from pregnant women. A breakup along these lines is important to correctly interpret the accuracy estimates.
Finally, even the day on which a sample is collected matters. There is growing evidence that the appearance of IgG antibodies in the blood happens late, Anantharaman, of the Rajiv Gandhi Centre of Biotechnology, said. Earlier this month, a preprint paper by US-based scientists reported that it took a median of 24 days from the onset of symptoms for high levels of IgG to show up in people with a mild form of COVID-19. The paper concludes that widespread antibody tests should ideally be carried out three to four weeks after people first begin to show symptoms.
However, Sapkal revealed that while most of the 75 samples for specificity had been collected 11 days after the onset of symptoms, the rest had been collected before this time, including on the very day the symptoms first showed.
Given such samples would have been negative for IgG, were such people retested at a later date? If so, how did the retesting affect the sample size? A follow-up question to Sapkal didn’t elicit a response.
To understand nuances like this, many researchers are calling for detailed data. Panda, of Ganit Labs, said the ICMR should release headline numbers like sensitivity and specificity as well as data about each individual patient, and whether sensitivity and specificity were tested in more than just one group of 150 people (75+75).
As it happens, even Zydus Cadila, the company tasked with manufacturing the kit, hasn’t published such details on its website. Multiple queries to the company asking for the product insert for the antibody kit – a document that typically contains this information – went unanswered. And the opacity doesn’t seem to be limited to Zydus Cadila.
India’s drug regulator, the Central Drugs Standards Authority, has approved some 102 COVID-19 diagnostic kits in India. It hasn’t shared any details of these kits on its website either – nor has ICMR, which has validated some 13 antibody kits.
This is not sensitive information. The list of all diagnostic kits the US Food and Drugs Administration approved is accompanied by documents explaining how each kit was validated.
Why transparency matters
There is a movement around the world towards greater transparency in health research, including diagnostic-kit research. In 2015, a global collaboration of scientists called the EQUATOR network published the Standards for Reporting of Diagnostic Accuracy Studies (STARD) guidelines, a checklist of items to be published alongside claims of diagnostic accuracy.
The 30 items on the list include validation study design, number of participants, how they were recruited and why a particular ‘gold standard’ was chosen (RT-PCR in ICMR’s case). In a paper accompanying the STARD guidelines, published in the British Medical Journal that year, the authors explained why diagnostic accuracy numbers were meaningless without context.
“Diagnostic accuracy is not a fixed property of a test. A test’s accuracy in identifying patients with the target condition typically varies between settings, patient groups and depending on prior testing. These sources of variation in diagnostic accuracy are relevant for those who want to apply the findings of a diagnostic accuracy study to answer a specific question about adopting the test in his or her environment.”
What question does ICMR’s kit answer?
According to the STARD guidelines, one can say if the test is accurate enough only if one knows what it will be used for. And we don’t know if the ICMR’s kits will be used at diagnostic centres, to tell individuals whether they have antibodies or not.
Some countries, such as Germany, have considered – controversially – issuing immunity passports. The logic is that people who have tested positive for antibodies specific to the novel coronavirus are likely to be immune, and so can return to work. A May 11 tweet by ICMR said the kit could be used “at any level of clinical setting, including in public health centres and hospitals”, as if to suggest it could be used with individuals as well, instead of just with populations.
When asked whether the kit would be used for purposes other than sero surveys, ICMR spokesperson Rajnikant Srivastava said it’s possible.
On National Technology Day, @ICMRDELHI is proud to announce the development of first indigenous Human IgG ELISA kit for #Covid_19 testing. The kit, developed in a month’s time, would help to study SARS CoV-2 IgG antibodies’ presence in the Indian population. #MakeInIndia pic.twitter.com/FZmC4uKj33
— ICMR (@ICMRDELHI) May 11, 2020
If so, will people getting the test be told that the result could be a false positive? And should a kit with a high false-positive rate even be used in low-prevalence regions?
The other context in which the ICMR kit will be used is in sero surveys. According to ICMR, sero-prevalence will be calculated for four types of places, ordered according to the number of RT-PCR-confirmed cases there. Experts said that while hard-hit cities like Mumbai could have a prevalence greater than 5% – making false positives less of a problem – fewer than 1% of the people in other places could be infected. And here, the kit is likely to identify more false than true positives, rendering the data problematic.
J.P. Muliyil, a Vellore-based epidemiologist who is on an ICMR-appointed research committee for COVID-19 disease surveillance, agreed that this is a real danger. However, he added that ICMR’s kit was much better than the kits imported from China.
“Remember where we are coming from. Some of the Chinese kits had a specificity of around 50%,” he said. And even with the ICMR’s kits, the researchers performing the survey could adjust for the potentially high false-positive rate. That an indigenously developed kit looked good after one round of validation gives him hope. “I am quite encouraged,” he said.
Note: Some typos in this article were fixed at 11:49 am on May 24, 2020. In addition, at 9:40 am on May 27, it was corrected to note that Premier Biotech is the distributor, not the maker, of the tests used in the Santa Clara sero survey.
Priyanka Pulla is a science writer.
The reporting for this article was supported by a grant from the Thakur Family Foundation. The foundation did not exercise any editorial control over the contents of the article.