Kiran Mazumdar Shaw, chairperson and managing director of Biocon Limited. Photo: Reuters.
On July 11, Biocon Limited announced that the Drugs Controller General of India approved its drug Itolizumab “for emergency use in India for the treatment of cytokine release syndrome (CRS) in moderate to severe acute respiratory distress syndrome (ARDS) due to COVID-19.
That evening, Biocon’s managing director Kiran Mazumdar Shaw said in a panel discussion on India Today‘s news channel that all of the patients in the treatment arm of the randomised controlled clinical trials (RCTs for short) survived. In contrast, one-third of the patients in the comparator standard-care arm died. At the time of writing this article, the full clinical trial data has not been released – neither in a press release nor as a pre-publication draft manuscript.
Now, CRS is a constellation of dramatic systemic symptoms caused by the rapid release of chemical substances called cytokines. It has been well described in cancer and transplant medicine, following treatment with a new class of drugs known as monoclonal antibodies. There is also a view that CRS and cytokine storms are probably different entities even though they’re often used interchangeably.
Itolizumab, a novel biologic drug, described as a humanised anti-CD6 monoclonal antibody developed by Biocon Biologics, has been approved in India for moderately severe chronic plaque psoriasis. As far as can be ascertained, the drug hasn’t been approved by the US Food and Drug Administration or the European Medicines Agency, quite possibly because Biocon hasn’t applied for such marketing authorisation. There is currently a phase 1b trial in progress in the US, to treat lupus and lupus nephritis.
Clinical trial of Itolizumab
Surprisingly, neither Biocon’s press release nor the India Today TV programme made any mention of the actual number of patients that participated in the clinical trial (see footnote). I have therefore relied on details provided in the submission that the investigators made to the Clinical Trials Registry of India.
Patients who tested positive for COVID-19, were over 18, with oxygen saturation in air of less than 95% and moderately severe ARDS were eligible to be enrolled as participants in the clinical trial. The randomisation method was not specified. The primary outcome measure was mortality at one month.
There was no mention in the trial protocol of the modelling techniques used to calculate the required sample size. The trial’s conductors did not state the background mortality rate or the minimum reduction in the baseline mortality rate that they though were clinically significant and therefore worth detecting. There was also no specification of the type 1 error or of the minimum power of the study considered desirable.
All of these calculations are necessary in this kind of trial to ensure the trial yields a statistically valid result that would be externally generalisable.
Nevertheless, the target sample size was 30 patients – to be randomised to the treatment arm and the control arm in a 2:1 ratio.
Also read: No COVID-19 Test Is 100% Right, so Their Errors and Results Are Both Important
Criticism of the trial
As a phase 2/3 RCT, this trial is seriously deficient in its design. Its defects – particularly the extremely small sample size – largely invalidate the claims made in the press release from Biocon, that “primary endpoints for reduction in mortality rate were met” or that “the data is compelling”.
The researchers have also not published data that would allow independent experts to assess their claim in the press release, that “Itolizumab demonstrated statistically significant advantage over the control arm, in one month mortality rate”.
With such data and without further larger trials, it is difficult to see how the company can fulfil its ambition to “take this therapy to other parts of the world impacted by the pandemic”.
The lack of a clearly stated randomisation plan raises doubts about whether the allocation of participants to the two groups was truly random. When the trial’s researchers report the trial’s outcomes in scientific paper to be published by a journal, it is difficult to see how they will meet the CONSORT Statement criteria. The better scientific journals use these criteria as a way to evaluate good practice, and those not meeting the criteria may not have their work published in the journal.
To understand the rationale for this assessment, we need to first understand the statistical basis of RCTs. This is a subject around which there is a large body of literature. The statistical considerations of a trial are so critical that many research teams insist on professional statistical advice and trial oversight.
Ethics committees often insist on such statistical inputs since a poorly designed study is, by definition, unethical. Medical journal editors often commission a separate statistical peer review of research papers that report such trials.
Say a condition has a mortality rate around 25% and a proposed new treatment offers the promise of reducing it by, say, an enviable 40%. In this case, it is very easy to conduct a small trial that achieves a reduction as big as this purely by chance. Indeed, the fact that Biocon’s chief medical adviser used the term ‘statistical significance’ in his statement shows that they understand the importance of conducting a statistically valid study. The greater the sample size, the less likely it becomes that the observed difference arose due to chance and the more likely it is that the results of the trial reflect the true situation.
The standard method in an RCT is to postulate a null hypothesis that the drug under study is really no better or worse than a placebo. The trial then collects data on the outcomes in the two groups to test the plausibility of this hypothesis.
This study is a classic case of what is known as an alpha or type 1 error in statistical hypothesis testing. A type 1 error is when you erroneously reject the null hypothesis when that hypothesis is in fact true. Put another way, it is the error when we observe a difference between the two groups in the trial when no such difference exists in the ‘population’ of similar patients.
Statisticians worry greatly about such errors and seek to ensure that a clinical trial is large enough to avoid such errors. They also worry about having a large enough sample to avoid a type 2 error, which arises when a real difference is missed.
There are tools available that allow you to calculate a sample size. Specifying a background mortality rate of 25% and an anticipated reduction due to the drug to 15% (i.e a 40% reduction in risk of death from baseline) and the usual 5% type 1 rate and 80% power, I get a minimum sample size of 500 if the allocation to the 2 arms is to be equal, or 576 if it is to be 2:1.
Also read: COVID-19: The Risks of Letting Technology Dictate Treatment
The irony is that Itolizumab may well be a breakthrough. If it is and if its real effects can be shown conclusively in a large trial, then Biocon would be able to boast of an unbeatable product. As things stand, it is unlikely to make that breakthrough without much greater rigour in the science of clinical trials. India’s pharmaceutical industry cannot become world-class on the back of shoddy clinical trials.
Footnote: Details were not available on the Biocon website at the time of writing this article, early on July 13. Biocon uploaded a press release later the same day. This footnote was added at 9 pm on July 14, 2020.
Dr Jammi Nagaraj Rao is a public health physician, independent researcher and epidemiologist in the UK.