A blood sample is held during an antibody testing program at the Hollymoor Ambulance Hub, Birmingham, June 5, 2020. Photo: Simon Dawson/Pool via Reuters.
Bengaluru: The Indian Council for Medical Research (ICMR) has said that it mistakenly overstated the accuracy of an antibody-testing kit manufactured by the Ahmedabad-based firm Zydus Cadila.
ICMR had developed the kit, called Covid Kavach Elisa, to detect past infections with the novel coronavirus, and tasked Zydus Cadila with manufacturing it. On May 14, the council published a press release claiming the kit had a specificity of 100% and sensitivity of 98.7% sensitivity. (Read our explainer of sensitivity and specificity here.) But according to a paper published in June in the Indian Journal of Medical Research, the kit’s specificity is only 97.9% and the sensitivity, 92.37%.
This drop in accuracy is significant because it could exaggerate ICMR’s estimates of how widely COVID-19 has spread.
Rajnikant Srivastava, head of ICMR’s research management, policy planning and coordination cell, told The Wire Science that the journal’s numbers were the correct ones, and that the numbers in the press release had been published by mistake.
According to Priya Abraham, director of the National Institute of Virology, Pune – which is the ICMR agency that developed the kit – the erroneous numbers referred to how closely the kit’s performance matched between two different laboratories.
These numbers have nothing to do with sensitivity and specificity, so it’s odd that they found themselves in a press release about the kit’s accuracy.
Surprising accuracy
In fact, the May 14 press release from ICMR was surprising for multiple reasons. First, it contradicted what government officials had said earlier. On May 11, health ministry joint secretary Lav Agarwal said during the day’s press briefing that the Covid Kavach Elisa kit had 97% specificity and 92% sensitivity. Four days later, the ICMR press release revised the numbers dramatically.
When The Wire Science asked Srivastava about this inconsistency then, he said both the joint secretary’s statement and the press release were correct. According to Srivastava, the lower accuracy numbers shared by Agarwal applied to the first round of validation of an initial version of the kit. The press release, on the other hand, referred to a second round of validation, of the finished product manufactured by Zydus Cadila. And between the two rounds, the kit had been tweaked to improve its accuracy, Srivastava said.
“Lav Agarwal was talking about the initial innovation. Now you have the product,” he wrote on an ICMR-administered WhatsApp group for journalists.
The Wire Science asked the same question to Gajanan Sapkal and Pragya Yadav, both NIV virologists at ICMR who developed the kit, over telephone on May 16. They (separately) confirmed this explanation and said the kit had been validated with 75 positive and 75 negative samples. They also said a paper on the validation methods would be published later. But multiple requests to Zydus Cadila in May for a manual describing the validation methods went unanswered.
Nevertheless, the claim of 100% specificity surprised many independent experts, given the dramatic improvement between the two rounds of validation. “It is very hard to reach 100% specificity,” said Srinivasaraman Govindarajan, a radiologist and director of the Chennai-based Anderson Diagnostics.
Paper contradicts press release
Then, earlier this month, ICMR finally published its paper in the Indian Journal of Medical Research describing how it validated the Covid Kavach Elisa kit. The authors, including Sapkal, Yadav and Abraham, wrote that they used a test called a micro-neutralisation assay [footnote]A test that detects neutralising antibodies to SARS-CoV-2 in the blood[/footnote] to determine whether blood samples from patients contained antibodies to the novel coronavirus. They then ran 131 such antibody-containing samples through the kit to arrive at a sensitivity of 92.37%. They also ran 382 antibody-less samples to arrive at a specificity of 97.9%.
At the same time, The Wire Science also accessed two other documents: the manual for Zydus Cadila’s kit and a May 25 invitation from ICMR to diagnostic companies asking them to submit proposals to manufacture the kits. Both these documents mention the same sensitivity and specificity – 92.1% and 97.7%, respectively – which are similar to, but not the same as, the numbers mentioned in ICMR’s paper.
When asked about the mismatch between the paper and the May 14 press release, NIV’s Abraham agreed the press release was incorrect. She clarified that the 98% and 100% figures in the press release referred to how well the results from the kits had lined up with each other when they were used in two laboratories.
“When we did our external validations in other labs for this new ELISA, the external labs reports were compared with our results on the same panel of sera. That just reflects on the inter-lab comparison but NOT on ELISA’s sensitivity and specificity,” she wrote in a WhatsApp message. “Somebody has taken the wrong indices of 98% and 100 %, which should be 92% and 97.7%.”
(ELISA stands for enzyme-linked immunosorbent assay, a test used to look for the presence of specific antibodies.)
Meanwhile, a spokesperson from Zydus Cadila told The Wire Science that the company itself had never claimed 100% specificity and 98% sensitivity. They shared a filing from Zydus Cadila to the Bombay Stock Exchange on May 10, which merely mentions “high sensitivity and specificity” sans any numbers.
Why these numbers matter
ICMR’s and Zydus Cadila’s claims of accuracy vis-à-vis the Covid Kavach Elisa kit will play a key role in interpreting the results of a just-completed seroprevalence survey[footnote]A survey to determine how many people have antibodies to the novel coronavirus, indicating these people have already been exposed to the virus[/footnote] to estimate COVID-19 spread in India. For this survey, ICMR used its kit to look for antibodies in blood extracted from over 26,000 people across 65 districts in the country.
These districts were split into four strata, ranging from those with zero confirmed COVID-19 cases to those with a high number of confirmed cases. In a press conference on June 11, ICMR director-general Balram Bhargava said that across these strata, the study had uncovered a seroprevalence of 0.73%. He didn’t present a stratum-wise breakdown, however. The study itself hasn’t been published yet.
Also read: Are the ICMR Antibody Kits Really as Accurate as It Claims?
The Wire Science had previously reported that if seroprevalence is lower than 5%, any kit with low specificity can drastically inflate the number of false positives [footnote]An incorrectly positive result for a person who doesn’t have antibodies[/footnote], yielding a distorted picture of how much COVID-19 has spread in the country. This means any specificity lower than 99% is “pretty useless in most seroprevalence studies,” according to Prabhat Jha, an epidemiologist at the University of Toronto.
These concerns are now amplified by the lower specificity that the new ICMR paper claims. For example, if only 2% of people in a village have been exposed to the novel coronavirus, the probability that a positive result on the test is correct – i.e. the positive predictive value – is only 47%. This is no better than tossing a coin, said Dr Shriprakash Kalantri, an internal medicine specialist at Mahatma Gandhi Institute of Medical Sciences, Sevagram, because both ‘heads’ and ‘tails’ have a positive predictive value of 50%.
Problems with the IJMR paper
Despite these problems, several experts welcomed the ICMR paper, saying that even 92.37% sensitivity and 97.9% specificity meant the Covid Kavach Elisa kit was better than many others currently in the market. However, they also pointed out issues with the paper itself.
For example, the paper claimed that the positive predictive value of the kit was 94.44%, while the negative predictive value [footnote]The chance that a negative result is correct[/footnote] is 98.14%. Jacob John, a professor of community medicine at Christian Medical College, Vellore, said such a claim “had no meaning”.
The reason for this is that the positive predictive value of a test is determined by the actual prevalence of the virus in a region. In contrast, the authors of the ICMR paper seem to have assumed the ratio of positive samples (131) to the total samples (513) used to validate the Covid Kavach Elisa test to be the prevalence, to calculate the positive predictive value. This ratio – 25.5% – is unlikely to apply to large parts of India, rendering the positive predictive value of 94.4% misleading, John explained.
Siddhartha Das, a physicist and postdoc at the Université Libre de Bruxelles, Belgium, had questions about a plot called the receiver-operator characteristic (ROC) curve, published in ICMR’s paper on the Covid Kavach Elisa kit. He noted that the shape of the curve matched exactly with the graph in the ICMR’s invitation to companies to manufacture the kit. This is odd because the ROC curve – which is used to estimate how accurate the kit is – is plotted based on a kit’s validation data. However, the ICMR paper used validation data from 513 samples while the invitation relied on a seemingly separate, and smaller, set of 251 samples. The two ROC curves should therefore be different, Das said, and yet they look identical.
John agreed that it was impossible for both ROC curves to match exactly, and that ICMR officials could have mistakenly used the same graph in both documents. But this raises questions about how reliable the numbers in the invitation document are. If ICMR did perform another round of validation using 251 samples, different from the 513 samples used in its published paper, the data from the former should have been analysed separately. Abraham did not respond to a question on the matching graphs.
Given the critical role the kit plays in the seroprevalence study, it’s important that its limitations are clearly understood, says experts. Once it’s published, the study’s findings will drive critical policy decisions on how much to open up the economy in various regions and what interventions to implement. Bhargava said in the June 11 press conference that, based on the seroprevalence data, the infection fatality rate in the study was a very-low 0.08%, a figure he called “good news”. However, if the kit has inflated the seropositivity in some regions, the infection fatality rate may not be so low after all.
Without accounting for the kit’s limitations, there is a real danger that big decisions could be taken on the back of faulty data.
Note: This article was updated at 8:57 am on June 12, 2020, to include Prabhat Jha’s comment.
Priyanka Pulla is a science writer.
The reporting for this article was supported by a grant from the Thakur Family Foundation. The foundation did not exercise any editorial control over the contents of the article.