Reporting Quality Assessment of COVID-19 Clinical Trials in China at the beginning of the Pandemic: An Empirical Study based on the CONSORT Statement

Patient and Public Involvement

No patient involved.

RCTs Selection

We searched the ChiCTR for the registered COVID-19 clinical trials in China between the outbreak and April 2, 2020. The ChiCTR was established in 2005, requiring all clinical trials (including early and late trials, trials of marketed or non-marketed products, randomized or non- randomized trials) to be registered. It was assigned by the Ministry of Health of China as the representative registry of China to join WHO ICTRP in 2007. Clinical trials could be searched there after having been registered in China.

Search Strategy

COVID-19 in Chinese and English were included in the Target disease section of the database filter to search for the clinical trials, without restrictions on registration status, year, study type, study phase, study design, recruiting status, gender and other options. The search strategy is summarized in Figure 1.

Inclusion and Exclusion Criteria

All clinical trials related to COVID-19 treatments were included in our study. Drug (Traditional Chinese medicine, biomedicine, chemical medicine) and non-drug (Tai Chi, acupuncture, Tui Na and other physical intervention or mental therapy) intervention researches were eligible for inclusion. Then any clinical trials that were (1) not COVID-19 treatment related, (2) unapproved by the ethics committee, (3) canceled by the investigator, or (4) non-RCT, were excluded.

Data Extraction

Microsoft Excel was applied to manage the titles and registration numbers obtained from the database. Two reviewers (Zy Z and ZC) read the title/abstract information for preliminary screening and full-text content for further screening. Next, the data from eligible studies were extracted and recorded in Excel independently. Any disagreements (the score difference on the same item from two reviewers is greater than 4) were settled through double-checking and discussing the detailed item information and the CONSORT requirements.

Quality Assessment

The reporting quality of included literatures was evaluated according to the CONSORT 2010 containing 25 items. Each item of the assessed article could score ranging from 0 to 10. Furthermore, score of each item was classified into three levels in our study: “0” (the item was not presented at all), “0~8” (the item was partially presented with limited details), and “8~10” (the item was presented with adequate details). For example, item sample size would be scored 8, if the RCT provided the sample size determination information such as calculation formula, or 9, if it provided determination information such as calculation procedure with brief explanations, or 10, if it provided determination information such as the calculation procedure, formula and detailed explanation of any interim analyses and stopping guidelines when applicable. Cochrane risk of bias tool was applied to analyze the risk of bias of included RCTs, and each RCT could divided into “Low risk of bias”, “High risk of bias” and “Unclear risk of bias” in accordance with the detailed criteria of Cochrane Handbook. For example, for the risk of bias of the random sequence generation, referring to the random number table or using a computer random number generator would be judged as low risk of bias, while referring to birth or admission date would as high risk of bias.

Two reviewers (Zy Z and ZC) was jointly trained on the scoring criteria of the CONSORT statement, and a pilot scoring test was conducted to ensure the consistency of assessment. The final score of each item was the average of the two reviewer’s scores. Similarly, the two reviewers assessed the risk of bias of included literatures, independently. Any disagreements would be settled through double checking and discussion.

Statistical Analysis

Descriptive analysis was applied for basic information of included literatures, as well as their evaluated results in accordance with the CONSORT statement and Cochrane handbook. And ANOVA was conducted to explore reporting quality difference of RCTs between intervention types (Drug VS Non-drug, and TCM VS Non-TCM) and primary sponsor locations. All analysis was performed in SPSS software, and a p value less than 0.05 was regarded as significant.

RCT Selection

Following preliminary screening with the title, abstract information and registration number of 578 clinical trials, 179 non-COVID-19 treatment records were excluded. Next, we checked the full-text of the remaining clinical trials, and removed 227 records for (1) non-approval by ethics committee (n=66), (2) cancellation by the investigator (n=50), (3) non-RCT design (n=111). Ultimately, 172 RCTs met the inclusion criteria and were included for reporting quality and risk of bias assessment. The detailed selection process is summarized Figure 1 Prisma 2009 flow-chart summary of RCT selection process.

Click to enlarge

Characteristics of the RCTs included

The characteristics of the RCTs (a summary of their basic information) are summarized in Table 1. All the RCTs identified in this review were registered between January and April 2020. The first registration, from a primary sponsor institution in Wuhan, Hubei Province, dated back to January 23, 2020. All the primary sponsors located in 4 municipalities directly under the central government (Beijing, Shanghai, Tianjin, Chongqing) and 16 Provinces. Among the 172 RCTs, three used drug and non-drug intervention jointly. As for Traditional Chinese Medicine (TCM), one RCT used both TCM

and non-TCM therapies jointly. The sample size ranged from 16 to 20,000, with most RCTs distributing between 20 to 300 in sample size. The trial duration ranged from 20 to 1081 days. One RCT was registered for phase I and II clinical trials, and one for both phase II and III clinical trials.

Reporting quality of the RCTs

All the 172 RCTs stated the following items: background and objectives, trial design, participants, interventions, outcomes, and registration. The problems related to the other CONSORT items were evident (insufficient explanation of how the sample size was determined)（n=0）; no mention of the statistical methods used, or when recruitment and follow- up started; inadequate presentation of the participants flow, numbers analyzed, baseline data, outcomes and estimation; and no reporting of ancillary analyses, harms, limitations, generalizability and interpretation. Randomization details were not presented.

Table 2 presents the summary of the proportion result. Only 93 RCTs (54.07%) met the CONSORT title/abstract requirement, identifying it as a randomized trial. There were significant discrepancies between five randomization items: the sequence generation (167, 97.10%), implementation (81, 47.09%), allocation concealment mechanism (25, 14.53%), and blinding (20, 11.63%). None presented the statistical methods. High protocol (94.77%) and funding (98.84%) reporting quality were noted.

The scoring results are summarized in Table 3 and their distribution presented in Table 4. The total score, mean score and standard deviation of the scoring system is 250, 70.01 and 9.74, respectively. The total scores of the RCTs ranged from 47.5 to 99.75. Although most dispersed between 68.4 and 78.85. Moreover, the top three items were:

the background and objectives (10.00), registration (10.00) and funding (9.90), while the lowest score all came from randomization part, including blinding (0.59), allocation concealment (0.88), and implementation (1.43), and sequence generation (2.71).

p<0.05 Intervention 1: A-Drug, B-Non-Drug Intervention 2: A-TCM, B-Non-TCM Location of the primary sponsor institution: A-4 municipalities directly under the central government, B- Hubei province, C- Other provinces. Table 5: Differences in reporting between intervention type and primary sponsor location.

Although the proportion of RCTs reporting the sequence generation exceeded 90%, the scoring result (2.71) revealed that more than 70% (129, 77.25%) of them only referred to the method (such as the computer random number generator, randomized number table, or hospital record number, etc.) used to generate the sequence without any restriction details. Of the remaining RCTs, only three provided detailed sequence generation information. For allocation concealment mechanism, 73 RCTs only referred to the sealed envelopes, case record number and other concealment without any descriptions of the steps taken to conceal the sequence. Among the 81 RCTs reporting on implementation, only four provided detailed (who generated the random allocation sequence, who enrolled participants and who assigned participants to interventions) information. Eleven RCTs only referred to the blinding without providing detailed statements.

ANOVA was conducted to test for significant differences between several subgroups. The result is summarized in Table 5.

The p values for interventions 1 (drug intervention vs non-drug intervention) and 2 (TCM vs Non-TCM intervention) were below the significant level (0.05), indicating significant difference between the drug and non-drug intervention scores as well as the TCM and non-TCM scores. Moreover, for the primary sponsor location, the difference between the locations was not significant.

The location of the RCTs was tested to judge the correlation between the location and RCT quality. We divided locations into three categories, Hubei province, 4 municipalities directly under the central government, and other provinces. The results showed that there was little correlation between location and RCT quality (p=0.244).

Bias Risk Assessment

The result of the risk of bias evaluation by the Cochrane Handbook is summarized in Table 6. A total of 130 RCTs had low risk of bias in random sequence generation, which relied on random number table, and computer random number generators. Seven used predictable methods such as record number and were defined as high risk of bias, and remaining 35 didn’t provide sufficient information concerning random sequence generation. Only 16 had low selection bias, 3 high selection bias, while the remaining 153 didn’t identify clear allocation concealment, only referring to using envelopes not the detailed procedures whether numbered, sealed or not, and were judged into unclear selection bias. Only 10 had low performance bias, meanwhile 68 had high performance bias. Similarly, 11 had low detection bias, and 68 had high performance bias. Besides, we also discovered the problems related to other types of bias, for example, monitoring details of physicians-recommended drugs were not available, which potentially caused performance bias.

Evaluation according to the CONSORT statement

Researchers are encouraged to use the word “randomized” in the study title to indicate that the patients were randomly assigned to the comparison groups [8]. Lacking a specific title impedes the report from being identified as a randomized trial. In our study, only 93

(54.07%) of the 172 RCTs identified the clinical trial as randomized (Table 1), which implies that 79 RCTs were not designed following the CONSORT statement.

Sufficient randomization is an effective measure to reduce selective deviation and guarantee the authenticity of the results [12]. In particular, a randomized trial needs to follow the randomization, control and blinding principles; however, the blinding and allocation concealment mechanism items were suboptimal among the RCTs in our study [13]. Those two items help not only a truly randomized implementation process but also prevent effect over estimation (especially in subjective outcome indicators such as pain, itching, and mood) as well as avoid implementation and measurement bias [14].

Our findings indicate insufficient emphasis on randomization and blinding which may cause high risk of bias and inaccurate results. We recommend that researchers should use the scientific random sequence generation methods, allocation concealment mechanism (described in detail), as well as the placebo control or blinding whenever possible. This will reduce the risk of bias, ensuring the feasibility and repeatability [15].

The Result items of the CONSORT checklist scored “0”, although it didn’t mean the quality of these items were low, for example, although the baseline data-the patients’ information table was not provided, the methods of collecting and managing baseline demographic and clinical characteristics were illustrated including Case Report Form (CRF) and Electronic Data Capture System (EDC), considering that the applicants would update these documents in ChiCTR in the future.

Factors Impacting Trial Quality

Drug RCTs show significantly higher quality than Non-drug RCTs (p<0.05) (Table.4). In general, masking is more difficult for participants and researchers because of the inapplicability of placebos in Non-drug trials, as demonstrated in a study related to continuous renal replacement therapy (CRTT) treatment [16, 17]. Also, some Non-drug trials’ interventions include subjective factors (for example, psychological support and cognitive behavioral therapy interventions) which were difficult to quantify and repeat [18].

TCM RCTs showed significantly lower quality than Non- TCM RCTs (p<0.05) (Table.5). Especially, scores of TCM RCTs in sequence generation, allocation concealment mechanism, implementation and blinding were lower those of Non-TCM RCTs. According to Mao et al., it is difficult to use simulated drugs of TCM, making it challenging to carry out blind method in TCM RCT studies [13]. Therefore, the CONSORT Method-item scores of TCM RCTs were generally lower than those of Non-TCM RCTs.

Reviewing former systematic studies of RCTs conducted in China, similar results of low reporting quality were found. One study, identified in PubMed and the Journal Series of the Chinese Medical Association, included Chinese RCTs and showed that 82.4% of the studies provided no blinding- related information, 64.8% failed to report the randomization methods, none mentioned the allocation concealment [19]. MIN Jie, et al. pointed that most Chinese RCTs reports, especially TCM RCTs from 2010, contained insufficient information, affecting quality. Recent studies in year of 2016 and 2019 also found poor quality of published RCTs in China, which was similar with the results of a review dated back to 1999, a review of Jin-Ling Tang et al., commented that “The quality of trials of traditional Chinese medicine must be improved urgently” (on page 161). Over two decades, some RCTs in China still remained low quality and high risk of bias, which need to be improved further [20, 21, 22, 23].

Furthermore, the gap between RCTs in China and RCTs in other countries for blinding and allocation concealment mechanism reporting, cannot be ignored. One study compared 137 Journals published RCTs (excluding Chinese RCTs), assessing the allocation concealment mechanism and blinding, and found that while 16.1% of the RCTs reported allocation concealment adequately and 43.0% reported blinding methods. Another study, showed that 30.0% (35 of 144) reported allocation concealment adequately and 16.0% (22 of 136) reported double-blinding [24, 25]. There does exist evident insufficiency in reporting quality of RCTs in China, comparing with other countries, which indicates that Chinese researchers should take more efforts to comply with the CONSORT, especially in allocation concealment mechanism and blinding methods.

In terms of content, some trials, not truly COVID-19 related (focusing on mood disorders and anxiety or other diseases instead of COVID-19), were reported as COVID-19 studies [26, 27]. Besides, different institutions also reported on the same treatment.

The CONSORT statement was designed for reporting trials, and the publication of completed COVID-19 RCTs may be prolonged. Considering the harmful effect of the pandemic, any updates of related studies are valued in the search for a cure. Yet, it is important for researchers to know the true quality of existing COVID-19 clinical trials. As a result, we selected the ChiCTR registered RCTs to profile the true reporting quality and risk of bias of the COVID-19 clinical trials currently conducted in China, contributing to improving the quality of trials design and pandemic response.

In summary, although numerous of COVID-19 clinical trials have been reported in China, the general result appears to be of low-quality. Moreover, fake COVID-19 clinical trials applications were also reported, not advancing the study of COVID-19 at all.

Limitations

The CONSORT statement is generally used to evaluate reporting quality of published trials and may present problems when used to evaluate the registration records. The public ChiCTR information was limited, making it difficult to assess some items of the CONSORT statement checklist. Besides, we did not contact the RCT applicants for explanations when dealing with vague risk of bias information during evaluation. Furthermore, following the RCT selection, the reviewers didn’t keep tracking them on the ChiCTR for potential updates, which may have biased the quality assessment result.