Original articlesWhat contributions do languages other than English make on the results of meta-analyses?
Introduction
Meta-analysts have little control over random errors but can exert at least some control over systematic ones. Including only a portion of all available evidence may introduce systematic errors into the review process and threaten its validity. Grégoire and colleagues reported that 78% of identified meta-analyses of randomized trials had language of publication restrictions [1]. The majority (93%) of these restrictions were at the expense of excluding trials published in languages other than English (OEL).
One way to evaluate whether language restrictions are a sensible policy for meta-analysts is to assess the quality of reports of randomized trials. Language restrictions might be appropriate if the quality of reports of OEL were different compared to English language (EL) trials. In a previous study, members of our group [2] compared the methodological characteristics and analytical approaches of 133 EL randomized trials published between 1989 and 1994, with reports of 96 randomized trials published in French, German, Italian, and Spanish over the same time period and type of journal. Within the same language the reports were assessed under masked conditions, using a scale developed with appropriate rigorous standards [3].
The differences found in this study, between OEL and EL trials in the quality of reporting, with respect to randomization, allocation concealment, double-blinding, dropouts and withdrawals, or overall total score, were neither statistically nor substantively significant. The mean differences in the quality of reporting between OEL and EL trials was 5% for the total score, and ranged from 0% to 4% for individual items. Similar results have recently been reported elsewhere [4]. However, these studies did not address whether the exclusion of OEL alters the statistical results of a meta-analysis. This study addresses that question.
Section snippets
Selection of meta-analyses
We selected our meta-analyses from a collection of 251 meta-analyses of randomized trials from a larger database (n = 455) of such studies [5]. The details of the search strategy used to identify and retrieve the meta-analyses included in this study are reported elsewhere 6, 7. Briefly, a refined MEDLINE search strategy identified the meta-analyses. This search was supplemented with a search of the Cochrane Database of Systematic Reviews (1996, Issue 1). Meta-analyses were eligible if they
Between meta-analyses comparisons
Of 251 meta-analyses reviewed, 79 met our inclusion criteria (Table 1 and Appendix B). Only 19 meta-analyses actually included OEL in their quantitative analysis. One of these meta-analyses was dropped from further analysis because it did not report any binary outcomes, leaving 18 for further analyses. The remaining meta-analyses either had no language restrictions but did not incorporate OEL into a quantitative analysis (n = 22), or explicitly excluded such trials from their study (n = 38).
Discussion
Our primary result indicated only a 2% difference (ROR = 0.98), on average, between the treatment estimates with versus without explicit restrictions on the language of publication of the trials included. Combined with the narrow confidence intervals (0.81–1.17) and several sensitivity analyses, these results suggest that it is unlikely that any important clinical differences were missed. These results provide empirical evidence regarding an important issue that meta-analysts and others have
Acknowledgements
We thank Alison Jones, Leah Le Page, and Michael Saginur for helping to coordinate the study. Alejandro R. Jadad, MD, is a National Health Research Scholar, Health Canada. This research was funded by the Medical Research Council of Canada, Grant # 3705.
References (31)
- et al.
Selecting the language of the publications included in a meta-analysisis there a tower of babel bias?
J Clin Epidemiol
(1995) - et al.
Completeness of reporting of trials published in languages other than EnglishImplications for the conduct and reporting of systematic reviews
Lancet
(1996) - et al.
Assessing the quality of reports of randomized clinical trialsis blinding necessary?
Control Clin Trials
(1996) - et al.
Language bias in randomised controlled trials published in English and German
Lancet
(1997) - et al.
Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses?
Lancet
(1998) - et al.
Validation of an index of the quality of review articles
J Clin Epidemiol
(1991) - et al.
Beta-blockade during and after myocardial infarctionan overview of the randomized trials
Prog Cardiovasc Dis
(1985) - et al.
Identification of meta-analysesThe need for standard terminology
Control Clin Trials
(1990) - et al.
Empirical evidence of bias in infertility researchOverestimation of treatment effect in crossover trials using pregnancy as the outcome measure
Fertil Steril
(1996) Does blinding of readers affect the results of meta-analyses?
Lancet
(1997)