Abstract
Background: In a 2003 paper in BMJ, the authors made the tongue-in-cheek observation that there are no randomized controlled trials (RCTs) of parachutes. This paper has been widely read, cited and used to argue that RCTs are impractical or unnecessary for some medical practices. We performed a study to identify and evaluate claims that a medical practice is akin to a parachute.
Methods: Using Google Scholar, we identified all citations to the 2003 paper. We searched for claims that a specific practice was akin to a parachute. For each practice, we identified the desired outcome of the practice, and searched Google Scholar and ClinicalTrials.gov for RCTs that were conducted, ongoing, halted, planned or unpublished.
Results: Of 822 articles citing the original paper, 35 (4.1%) argued that a medical practice was akin to a parachute. Eighteen of the 35 (51%) concerned mortality or live birth, and 17 (49%) concerned a lesser outcome. For 22 practices (63%), we identified 1 or more RCTs: in 6 cases (27%), the trials showed a statistically significant benefit of the practice; in 5 (23%), the trials rejected the practice; in 5 (23%), the trials had mixed results; in 2 (9%), the trials were halted; and in 4 (18%), the trials were ongoing. Effect size was calculated for 5 of the 6 practices for which RCTs gave positive results, and the absolute risk reduction ranged from 11% to 30.8%, corresponding to a number needed to treat of 3-9.
Interpretation: Although there is widespread interest regarding the BMJ paper arguing that randomized trials are not necessary for practices of clear benefit, there are few analogies in medicine. Most parachute analogies in medicine are inappropriate, incorrect or misused.
In a widely cited 2003 BMJ article, the authors made the tongue-in-cheek observation that there are no randomized trials of parachutes.1 In an era in which proponents of evidence-based medicine increasingly rely on randomized controlled trials (RCTs) to show treatment efficacy, Smith and Pell1 argued that some medical practices are so beneficial that it would be silly to subject them to an RCT. The use of a parachute during free fall, such as a controlled jump from an airplane, is an example. Without the use of a parachute, the chance of death approaches nearly 100%, although there are scattered case reports of people surviving such a fall.2 With a parachute, the risk of death decreases dramatically, with recent estimates of 1.1 deaths per 100 000 jumps, a rate of 0.0011%.3 Of course, there are several limits to the parachute analogy in medicine. The first difference is etiology. Falling from an airplane has only 1 causal pathway leading to harm. In contrast, most human diseases have multifactorial etiologies,4 and any 1 practice may be unlikely to single-handedly reverse the outcome. The second limitation is the effect size. Parachutes improve survival from nearly 0% to nearly 100%. Empirical analyses show that few medical practices offer so large a magnitude of benefit. For instance, in a review of over 80 000 medical practices from the Cochrane database, Pereira and colleagues5 found only 1 medical practice that reliably had a large effect on overall mortality, and the absolute risk reduction (ARR) of this practice was about 33%.6
Despite the limits to the parachute analogy, the article by Smith and Pell1 has gained popularity in the medical community and is often used to criticize the need for RCTs of a specific practice. For instance, in a 2016 update to guidelines put forth by the US Department of Agriculture and Department of Health and Human Services, daily flossing was no longer recommended, as it lacked rigorous data showing benefit.7 In opposition, Holmes8 argued that long-term flossing was akin to a parachute and may not ethically be tested in randomized fashion. Whether daily flossing is a parachute may be debated, and the analogy may be exaggerated. We performed a study to determine how often researchers claim their medical practice is a parachute when it is in fact no such thing.
Methods
We identified on Google Scholar all articles that cite the paper by Smith and Pell.1 Our search was conducted from Jan. 1 to Mar. 31, 2016. We selected Google Scholar because of its extensive citation network and coverage,9 which are known to be superior compared to other citation engines, particularly its citation analysis.10,11 One reviewer (M.J.H.) screened titles and abstracts of identified articles for full-text review, excluding all articles in languages other than English, published in non-peer-reviewed journals or not related to human clinical medicine (e.g., veterinary medicine). Selected full-text articles were individually reviewed in full by 2 authors (M.J.H. and V.P); as there were no disagreements, all subsequent analyses were based on the reviews of 1 reviewer (M.J.H.). We included articles that used the paper by Smith and Pell1 to argue that a specific medical practice was like a parachute in that it could not ethically or practically be tested in an RCT. We did not include articles that used the paper to argue against the principles of evidence-based medicine in general. We defined a medical practice as any medication, procedure or system-based change intended to help prevent or treat a medical condition.
For all included articles, we searched Medline for RCTs or systematic reviews of RCTs that had investigated the subject referenced by the article. If multiple large RCTs were readily identifiable, they were each included in our analysis. A detailed search strategy for these RCTs is given in Appendix 1 (available at www.cmajopen.ca/content/6/1/E31/suppl/DC1), and all articles selected were reviewed by M.J.H. and V.P. For these RCTs, we ascertained whether studies gave a positive result (net benefit of the proposed practice), negative result (no benefit or net harm of the proposed practice) or mixed result (inconsistent evidence supporting the use of the proposed practice). For practices for which no RCT could be found, we searched ClinicalTrials.gov to identify any ongoing or unpublished RCTs available. For studies with no published or unpublished RCTs, we summarized the nature of the practice, and M.J.H. and V.P. determined whether an RCT was clinically feasible.
Statistical analysis
We performed descriptive statistics. We calculated the ARR by subtracting the risk in the control arm from that in the experimental arm at whatever time point was reported by the author.
Ethics approval
This study of published reports did not require institutional board review approval. A protocol can be requested from the corresponding author.
Results
At the time of our investigation, the article by Smith and Pell1 had 822 citations on Google Scholar. Of the 822 articles, 35 (4.3%) directly compared a medical practice to a parachute or used the parachute argument to defend implementation of that practice (Figure 1). These were included in subsequent analysis. We did not identify any prior citation analyses for the article by Smith and Pell.1
Flow chart showing selection of articles on medical practices analogized to parachutes.
Of the 35 medical practices, 22 (63%) involved RCTs that were completed, ongoing, halted or planned (Table 1). Examples include stenting for renal artery stenosis, insulin analogues for the treatment of type 1 diabetes, simulation training for providers working in critical care medicine and metastasectomy for isolated pulmonary metastatic colorectal cancer. In 6 cases (27%), RCTs showed a statistically significant benefit of the practice; in 5 (23%), RCTs rejected the practice; in 5 (23%), RCTs gave mixed results; in 2 (9%), RCTs were halted; and in 4 (18%), RCTs were ongoing. If mixed data were accepted as showing a significant benefit, 11 practices (50%) were supported by identified RCTs.
The remaining 13 medical practices (37%) had not been tested in an RCT (Table 2). Several are long-standing practices, including mechanical ventilation for acute lung injury, emergency airway management and surgical management of acute epidural hematoma; others are recent practices, such as infection-control strategies for the prevention of perioperative skin and soft-tissue infections, perispinal etanercept administration for poststroke neurologic dysfunction and repair strategies for defective dental restorations.
Mortality or live birth was the clinical outcome of interest for 18/35 practices (51%). A lesser outcome, including infection control, reduced diabetic complications, cardiovascular events, clearance of parasites, pain or discomfort, weight gain, lead migration and effective dental restoration, was the outcome of interest for the remaining 17 practices (49%) (Figure 2).
Outcome of interest for medical practices analogized to parachutes.
Among the 6 practices with positive evidence in RCTs, the ARR could be calculated for 5 and ranged from 11% to 30.8%, corresponding to a number needed to treat (NNT) of 3-9. Notably, 3 of the 5 studies concerned mortality or live birth, and 2 reported the ARR for other outcomes.
Interpretation
Over a decade after publication of the article by Smith and Pell,1 which concluded that RCTs of parachutes or other highly effective practices would be ludicrous, we found few papers citing the article that argued that a medical practice is a parachute. Among those practices, only half referred to a practice whose clinical outcome was mortality or live birth. Of identified practices, more than half have been tested in an RCT, which undermines the claim that the practice is a parachute. Among the remaining practices, RCTs seem possible, even desirable, for several.
In this respect, our findings are similar to those of other empirical analyses. Glasziou and colleagues73 compiled a list of 16 examples of treatments that are universally considered beneficial and that lack randomized study. Djulbegovic74 extended this list to nearly 50 examples. Yet what must be acknowledged is that this set of interventions is a tiny fraction of all medical practices: as there are at least 80 000 practices,5 50 practices account for just 0.06% of medical interventions.
The proportion of purported parachutes for which RCTs gave a positive result in our study, 50% if mixed trial results are interpreted as positive, is similar to the reported rate of trials with positive results in the setting of genuine therapeutic uncertainty, just over 50%.75 This suggests that analogizing a medical practice to a parachute is done for practices that are, on average, no more likely to be beneficial than a typical medical practice tested in randomized fashion.
Moreover, previous parachutes in medicine have been shown to be overstated. The philosopher of science John Worrall commented, after listing several medical treatments, including appendectomy for acute appendicitis, "no RCT has ever been performed on any of these treatments and none presumably ever will."76 Yet there are now 4 RCTs of appendectomy versus antibiotics,77-80 which suggests that there may be a subset of patients who can be spared surgery. Another example is precision oncology. Experts have claimed that the use of next-generation sequencing to pair patients with cancer with targeted therapies is a medical practice of such great promise that RCTs are unethical.81 To date, 1 RCT has been conducted for this practice, yielding negative results.82 In both of these cases, the presence of a RCT, particularly one that gives a negative result, undermines the parachute analogy.
In cases in which the magnitude of benefit was estimable, we found NNT values of 3-9 and ARR values of 11%-30.8%. These gains are smaller than those with parachutes, which have ARR values greater than 99% and NNT values approaching 1. These results suggest that, even when RCTs support the use of the practice, the use of the parachute analogy is inappropriate.
Although more RCTs are being performed per annum than ever before,83 the idea of the RCT as the pinnacle of evidence-based medicine has been criticized.84 Moreover, there is growing interest in the use of observational data, including big data and real-world data, to make causal inferences about the efficacy of novel treatments.85,86 One justification for this interest is that it may not be feasible or ethical to conduct RCTs for highly promising medical practices. Our results provide a reassuring note. Few medical practices have large treatment effects,5 and even practices believed to be parachutes often are not.
Limitations
Although we performed an exhaustive search of references to the seminal paper by Smith and Pell1 using a search engine with the widest citation network,10,11 we may not have captured all instances in which researchers likened a particular practice to a parachute, as many such instances may not have been captured by the use of a single search engine or may have predated the 2003 paper.
Second, and notably, only a small proportion of papers citing the article by Smith and Pell1 drew a specific comparison to a medical practice. This is largely in part because many researchers cite the paper to criticize generally the importance of RCTs. Nevertheless, this fact is also noteworthy. In over a decade since publication of the article, and although it has generated widespread discussion and interest, few citing papers argue that a practice in medicine is akin to a parachute.
Third, our paper does not imply that RCTs are always feasible, possible, necessary or ethical. We tried to be as objective as possible in our determination of which practices could be tested in RCTs in the future, but we acknowledge the subjective nature of this assessment and the inherent challenges of performing rigorous RCTs. In fact, it is inevitable that there will be situations in medicine in which decisions have to be made in the absence of randomized data.87 Moreover, as noted by Djulbegovic,74 there are indeed examples of practices universally thought beneficial in the absence of RCTs. However, our investigation provides further evidence that the number of such practices is few. We also provide a cautionary note: a researcher's belief that an intervention is a parachute seems a poor predictor of actual parachute practices.
Conclusion
Although there is widespread interest in the idea that some medical practices are like parachutes - with a magnitude of benefit so large and obvious that RCTs are unnecessary - few biomedical authors compare a specific medical practice to a parachute. When they do, over half refer to a practice that has been tested with an RCT, and half refer to an outcome of lesser importance than overall survival, findings that undermine the claim that the practice is a parachute. When RCTs have been conducted and estimate effect sizes, practices analogized to parachutes have ARR values that are smaller and NNT values that are larger than those for parachutes. Although we found that the parachute analogy is seldom used to describe a medical practice, when it is used it is often inappropriate, incorrect or misused.
Supplemental information
For reviewer comments and the original submission of this manuscript, please see www.cmajopen.ca/content/6/1/E31/suppl/DC1.
Footnotes
Competing interests: Sham Mailankody has received honoraria from Wedbush PacGrow for speaking at the annual health care conference and from PleXus Communications for continuing medical education activity. Vinay Prasad is funded by the Laura and John Arnold Foundation. No other competing interests were declared.
Contributors: Vinay Prasad conceived and designed the study. Michael Hayes acquired and analyzed the data. Victoria Kaestner contributed to data acquisition and assembly. Sham Mailankody and Victoria Kaestner contributed to data analysis and interpretation. Vinay Prasad and Michael Hayes drafted the manuscript. All of the authors revised the work for important intellectual content, gave final approval of the version to be published and agreed to be accountable for all aspects of the work.
References
- Copyright 2018, Joule Inc. or its licensors
In this issue
Article tools
Related Articles
Cited By...
- No citing articles found.