Abstract
Background: The quality of case reports, which are often the first reported evidence for a disease, may be negatively affected by a rush to publication early in a pandemic. We aimed to determine the completeness of reporting (COR) for case reports published on coronavirus disease 2019 (COVID-19).
Methods: We conducted a systematic search of the PubMed database for all single-patient case reports of confirmed COVID-19 published from Jan. 1 to Apr. 24, 2020. All included case reports were assessed for adherence to the CARE (Case Report) 31-item checklist, which was used to create a composite COR score. The primary outcome was the mean COR score assessed by 2 independent raters. Secondary outcomes included whether there was a change in overall COR score with certain publication factors (e.g., publication date) and whether there was a linear relation between COR and citation count and between COR scores and social media attention.
Results: Our search identified 196 studies that were published in 114 unique journals. We found that the overall mean COR score was 54.4%. No one case report included all of the 31 CARE checklist items. There was no significant correlation between COR with either citation count or social media attention.
Interpretation: We found that the overall COR for case reports on COVID-19 was poor. We suggest that journals adopt common case-reporting standards to improve reporting quality.
The coronavirus disease 2019 (COVID-19) pandemic demands rapid access to high-quality evidence. Case reports often provide the first published evidence for a disease, as they can detect novelties, generate hypotheses for future research, and offer insight into rare exposures and outcomes.1–4 In the early stages of an emerging infectious disease, case reports become integral pieces of evidence for interim planning and treatment decisions while higher-quality observational and interventional studies are pending.4–6 For example, case reports and case registries formed an important cornerstone in the delivery of information during the 2009–2010 influenza A pandemic. 5 There are concerns, however, that a rush to publication will result in poorer research quality during the current COVID-19 pandemic.7–9
One area that may be adversely affected by this rush is the quality of reporting in case reports. As with all research, case reports should meet strict requirements for completeness of reporting (COR) to ensure access to relevant information.10,11 Specifically, the CARE (Case Report) statement and checklist, a guideline that indicates best practices in case reporting, 11 outlines the relevant information to ensure comprehensive COR in case reports. The most recent version, updated in 2016, covers 14 topic areas with a total of 31 checklist items.10 Several studies across a variety of domains and journals have evaluated COR for case reports by calculating a composite score of the number of CARE checklist items that were complete.12–16
Our objective was to determine the COR of case reports related to COVID-19. We hypothesized that the overall COR score would be lower than the “acceptable” threshold of 75%, which is the average COR based on a previous study.12 The secondary objectives involved determining whether COR was affected by publication date, publication status, open-access status, and journal policies regarding CARE and publication ethics.
Methods
We performed a meta-epidemiologic study of COR among case reports published early in the COVID-19 pandemic. Reporting of our findings followed recommendations for the reporting of meta-epidemiologic methodology research.17
Definitions
According to the original CARE guideline publication, a case report is a “detailed narrative that describes, for medical, scientific, or educational purposes, a medical problem experienced by one or several patients.”11 To operationalize this definition, we considered “case reports” as any publications with clinical information from a single patient, which included imaging reports, problem-solving cases and correspondence. Similar to a previous study,12 we included only case reports with clinical data from only 1 patient, as the CARE checklist does not allow for evaluation of clinical information from more than 1 patient.
Study selection
We conducted a systematic search using a modified strategy proposed by the Centers for Disease Control and Prevention search recommendation for COVID-19. Specifically, we used the COVID-19 PubMed search alert filter to capture the latest case reports on COVID-19.18 The full search is presented in Appendix 1, available at www.cmajopen.ca/content/9/1/E295/suppl/DC1. We executed the search in PubMed for all case reports published from Jan. 1, 2020, to Apr. 24, 2020, on Dec. 9, 2020. Two authors (M.A.S. and J.L.) performed an initial screen of the abstracts in duplicate and independently. The search was reviewed by a librarian at our institution. We retrieved full texts for all included abstracts. Two reviewers (M.A.S. and N.G.) independently conducted the full-text review in duplicate.
We included studies if they were case reports of a single patient diagnosed with COVID-19 that were published online from Jan. 1, 2020, to Apr. 24, 2020. We excluded reports with an unconfirmed diagnosis of COVID-19, secondary studies (e.g., editorials without clinical data), studies with full texts that could not be obtained or were unavailable in English, studies without a digital object identifier (DOI), animal studies and studies presenting clinical information for more than 1 patient. Studies without DOIs were excluded to ensure there was a reliable identifier for secondary data collection.
CARE guideline
The CARE checklist, initially developed in 2013 using a Delphi process to ensure COR for case reports,11 was subsequently updated in 2016 to include a 14th topic, making a total of 31 checklist items.10 We used this version for our evaluation and identified the following sections as particularly relevant to clinical practice in COVID-19: patient information, clinical findings, diagnostic assessment, therapeutic intervention, and follow-up and outcomes.
Data extraction
Six authors (J.L., R.B., Y.V., K.E., G.M.D. and A.P.), who divided up the abstraction among themselves, developed and used a standardized form to extract the following bibliographic data: date of publication; publication status (journal preprint, postprint or published), as was determined using the journal website; and whether the journal explicitly endorsed either the CARE guideline or the EQUATOR (Enhancing the Quality and Transparency of Health Research) Network on the “For Authors” sections of their website.
Two authors (M.A.S. and N.G.), who both had graduate-level training in research appraisal, reviewed each case report using the CARE checklist. A pilot data extraction was conducted on case reports unrelated to COVID-19. The 2 authors then independently reviewed each case report in duplicate for CARE checklist items, following recommendations from the CARE guideline.10 Using institutional access, all journal texts were downloaded on Dec. 9, 2020.
Bibliographic analysis
All bibliographic data were obtained on Jan. 9, 2021. For information on open-access status, we used the Simple Query Tool from Unpaywall (Our Research), a publicly accessible platform that indexes articles’ open-access information.19,20 For citation count, we used institutional access to the Web of Science (Clarivate).21 For social media attention, we used Altmetric Explorer (Digital Science), which tracks mentions of the publications across online sites and social media.22 For journal information, we used institutional access to Ulrich’s Periodicals Directory (ProQuest LLC), a database that provides information on academic publications.23 For journal impact factor and total citation count, we used institutional access to the InCites Journal Citation Reports system, which is a subsection of Web of Science.24
Outcome measures
The primary outcome was the overall COR score for case reports related to COVID-19, with a focus on clinically relevant items. Secondary outcomes included change in overall COR score with publication factors (e.g., publication date and open-access status) and correlation between COR and citation count and between COR scores and social media attention.
Statistical analysis
We presented all data as means and standard deviations (SDs). According to a previous study,12 the COR score was calculated using the following equation:
The mean COR score was calculated using assessments from both raters. To ensure transparency in reporting,25 we also calculated a mean COR score for all items of the CARE statement.
Interrater agreement for independent assessments was assessed for the overall COR scores using the intraclass correlation coefficient (ICC)2,1 (2-way random-effects model for average measures [the average of 2 raters’ scores]).
To evaluate a difference in the mean COR scores with the mean COR score of 75% from the literature,12 we used a 1-sample t test. To evaluate a difference in the mean COR scores with the variables of CARE endorsement and open-access status, we used an independent-samples t test. To determine whether there was a linear relation between mean COR scores with Altmetric Attention Score and with article citation count, we used Pearson correlation coefficient.
All p values are 2-sided without correction for multiplicity of tests. We considered an α < 0.05 indicative of statistical significance. All statistical analyses were completed in SPSS version 25 (IBM).
Ethics approval
We obtained an exemption from the research ethics board at our institution, St. Michael’s Hospital, as there were no primary patient data involved.
Results
Of the 400 studies found in our initial search, we included 196 case reports that were published in 114 different journals. The bibliographic characteristics of the included studies are summarized in Table 1. A detailed summary for each study is provided in Appendix 2, available at www.cmajopen.ca/content/9/1/E295/suppl/DC1.
Bibliometric characteristics of included case reports
The included case reports were published between Jan. 4 and Apr. 23, 2020. A summary with month of publication for the included case reports and corresponding mean COR score is provided in Figure 1.
Search flow for case reports related to coronavirus disease 2019 (COVID-19). Note: DOI = digital object identifier. *Case reports could fit into more than 1 exclusion category.
Primary outcome
The mean COR score from the 2 raters was 54.4% (SD 16.3%), which was significantly less than the average COR threshold of 75% (t95 = −7.8, p < 0.001) (Figure 2). The ICC2,1 for the overall COR score was 0.81 (95% confidence interval 0.74–0.85), which indicates good agreement.
Number of case reports published on coronavirus disease 2019 (COVID-19) from the beginning of 2020 to the date of the search (Apr. 24, 2020) with corresponding mean completeness of reporting (COR) scores. The error bars represent the standard deviation.
No one case report included all of the 31 CARE checklist items (Table 2). The 4 clinically relevant items with the highest COR scores were “Main symptoms of the patient (chief complaints)” (mean COR 96%, SD 1.4%), “Diagnostic methods” (mean COR 98%, SD 0.4%), “Types of intervention” (mean COR 84%, SD 2.2%) and “Diagnostic reasoning” (mean COR 76%, SD 25.0%).
Completeness of reporting scores for each CARE topic and question11
The 4 clinically relevant items with the lowest COR scores were “Diagnostic challenges” (mean COR 13%, SD 0.4%), “Prognostic characteristics (e.g., staging in oncology)” (mean COR 14%, SD 6.9%), “Intervention adherence and tolerability” (mean COR 16%, SD 14.0%) and “Adverse and unanticipated events” (mean COR 22%, SD 10.4%).
Secondary outcomes
We did not find significant differences in mean COR scores for the following analyses: articles published in the 2-month period of January–February compared with March–April (48.1% v. 55.1%, respectively; t194 = 1.7, p = 0.08); journals that endorsed the CARE guideline compared with those that did not (52.4% v. 55.5%, respectively; t194 = −1.3, p = 0.2); open access articles compared with those that were not (54.4% v. 55.1%, respectively; t194 = −0.1, p = 0.7); and articles published in open access journals compared with those that were not (55.2% v. 54.2%, respectively; t194 = 0.4, p = 0.7).
There was no significant linear relation in mean COR score with Altmetric Attention Score (R = 0.016, p = 0.8) or with article citation count (R = −0.002, p > 0.9). There were 10 studies for which Altmetric Attention Score was not available and 17 studies for which citation count was not available.
Interpretation
We identified 196 case reports on COVID-19 published in 114 different journals from January to April 2020. Overall, the COR was poor, with a mean COR score of 54.4%. No single case report contained all the items in the CARE guideline. Furthermore, the clinically relevant items that were most poorly reported pertained to diagnostic challenges, prognostic characteristics, therapeutic adherence and tolerability, and adverse and unanticipated events. No other factors significantly affected COR, and there was no correlation of COR with either social media attention or article citation count.
Although classically considered low-quality evidence,4 case reports become critically important during the early stages of pandemics.6 During the 2009 H1N1 pandemic, for example, case reports and other observational studies were among the earliest types of evidence available to inform clinical practice and facilitate knowledge dissemination to the global community. 5,26,27 Specifically, these case descriptions aided the early understanding of the disease,26 as they informed clinical presentation, epidemiologic characteristics and outcomes. This is illustrated by a bibliometric analysis of literature from the Middle East respiratory syndrome (MERS) outbreak, which found that case reports were among the top 10 cited publications from this period.28 Furthermore, case reports serve as the foundation for higher-quality follow-up studies with larger samples.3 In short, case reporting is a highly flexible medium that can be strategically pivoted during disease outbreaks, and it frequently forms the “first line of evidence.”29
Our findings, however, show that this front line can be easily undermined with suboptimal reporting. In particular, suboptimal reporting in the clinically relevant areas of intervention adherence, tolerability and adverse events exacerbates the concern around decision-making regarding unproven therapies in critically ill patients.3,6 The literature shows that incomplete reporting is not unique to the current COVID-19 pandemic. A report by the World Health Organization on the H1N1 pandemic,26 for example, found that important components of case records were incomplete, which negatively affected the lines of communication.
We also found that COR did not correlate with either article citation or social media attention. One interpretation is that readers do not preferentially consider quality of reporting when citing or discussing case reports. As a result, we believe that ensuring adequate reporting is best accomplished through endorsement of reporting guidelines by each journal. This has been demonstrated for more robust forms of evidence, such as randomized controlled trials (RCTs), in which journal endorsement of Consolidated Standards of Reporting Trials (CONSORT) improved overall COR.30 In terms of case reports, 1 study found that endorsement of the CARE guideline among 4 high-impact medical journals improved COR.12
Journal endorsement alone, however, may prove insufficient. Among RCTs, for example, reporting quality was still suboptimal even with CONSORT endorsement, with the authors suggesting that journals should implement stronger measures to ensure COR.30 We suggest that journals employ large-scale adoption of standardized documentation to improve the completeness of case reporting.27 This approach lent itself especially well to pandemics in the case of registries. Standardization of case report forms across registries during the early stages of the 2009 H1N1 pandemic improved the ability of standardized case reporting forms to support future research,5 and standardization of forms was also shown to be globally feasible.31
Limitations
Our study has several limitations. First, we included only single-patient case reports, which restricted the generalizability of our findings to this specific type of study. Second, we excluded all studies with full texts that were not published in English and those without a DOI, which may have introduced a language bias or a bias against journals that do not assign a DOI. Third, the CARE guideline is intended for clinical reports, which may limit its generalizability to imaging reports and correspondence. This may have led to a lower overall COR score owing to size restrictions. Fourth, although our study was intended to investigate the reporting characteristics of case reports in the initial months of the COVID-19 pandemic, we acknowledge that publication characteristics, such as COR and citation count, may have changed with time. Fifth, our search was restricted to only 1 database (i.e., PubMed) and did not include preprint servers for case reports, which may have introduced publication bias. Furthermore, the search filter used for COVID-19 case reports was not validated at the time of the study. Sixth, we did not independently verify the validity of the data from the databases used for our secondary outcome measures. Finally, the use of the CARE guideline to assess COR, though previously reported,12–14,16 imputed a degree of subjectivity, as the recommendations do not specify the depth or explicitness of completeness required for each topic. Nevertheless, the overall interrater agreement of the COR scores suggests good reliability between the 2 raters.
Conclusion
The overall reporting quality of COVID-19 case reports was poor. We suggest the wide-scale adoption of a common standard for case reporting by journals. Future studies should comprehensively investigate the quality of reporting among other types of evidence for COVID-19 and explore the reasons for suboptimal COR, if it exists.
Acknowledgement
The authors thank Altmetric for providing data for this study free of charge for research purposes.
Footnotes
Competing interests: Rishad Khan has received research grants from AbbVie and Ferring Pharmaceuticals, and research funding from Pendopharm. Samir Grover has received research grants and personal fees from AbbVie; has received personal fees from Takeda, Amgen and Lupin; and has equity in Volo Healthcare. No other competing interests were declared.
This article has been peer reviewed.
Contributors: Michael Scaffidi and Nikko Gimpaya conceived and designed the study, and acquired and analyzed the data. Juana Li, Yash Verma, Karam Elsolh, Rishi Bansal, Gemma Donn and Abhishek Panjwani contributed to data acquisition and assembly. Samir Grover and Rishad Khan contributed to data analysis and interpretation. Michael Scaffidi and Nikko Gimpaya drafted the manuscript. All of the authors revised the work for important intellectual content, gave final approval of the version to be published and agreed to be accountable for all aspects of the work. Michael Scaffidi and Nikko Gimpaya contributed equally for first authorship.
Funding: No funding organization had any role in the design and conduct of the study, collection, management, analysis and interpretation of the data, and preparation, review or approval of the manuscript.
Data sharing: Study data are available on request.
Supplemental information: For reviewer comments and the original submission of this manuscript, please see www.cmajopen.ca/content/9/1/E295/suppl/DC1.
This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BYNCND 4.0) licence, which permits use, distribution and reproduction in any medium, provided that the original publication is properly cited, the use is noncommercial (i.e., research or educational use), and no modifications or adaptations are made. See: https://creativecommons.org/licenses/by-nc-nd/4.0/
References
- © 2021 Joule Inc. or its licensors
In this issue
Article tools
Related Articles
Cited By...
- No citing articles found.