Methods of Systematic Reviews and Meta-Analysis
AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews

https://doi.org/10.1016/j.jclinepi.2008.10.009Get rights and content

Abstract

Objective

Our purpose was to measure the agreement, reliability, construct validity, and feasibility of a measurement tool to assess systematic reviews (AMSTAR).

Study Design and Setting

We randomly selected 30 systematic reviews from a database. Each was assessed by two reviewers using: (1) the enhanced quality assessment questionnaire (Overview of Quality Assessment Questionnaire [OQAQ]); (2) Sacks' instrument; and (3) our newly developed measurement tool (AMSTAR). We report on reliability (interobserver kappas of the 11 AMSTAR items), intraclass correlation coefficients (ICCs) of the sum scores, construct validity (ICCs of the sum scores of AMSTAR compared with those of other instruments), and completion times.

Results

The interrater agreement of the individual items of AMSTAR was substantial with a mean kappa of 0.70 (95% confidence interval [CI]: 0.57, 0.83) (range: 0.38–1.0). Kappas recorded for the other instruments were 0.63 (95% CI: 0.38, 0.78) for enhanced OQAQ and 0.40 (95% CI: 0.29, 0.50) for the Sacks' instrument. The ICC of the total score for AMSTAR was 0.84 (95% CI: 0.65, 0.92) compared with 0.91 (95% CI: 0.82, 0.96) for OQAQ and 0.86 (95% CI: 0.71, 0.94) for the Sacks' instrument. AMSTAR proved easy to apply, each review taking about 15 minutes to complete.

Conclusions

AMSTAR has good agreement, reliability, construct validity, and feasibility. These findings need confirmation by a broader range of assessors and a more diverse range of reviews.

Section snippets

Background

What is new?

  • AMSTAR, a new instrument for evaluating systematic reviews, is reliable, valid, and easy to use.

  • Currently, there is no agreement on which instrument to use when measuring the quality of systematic reviews. AMSTAR, a development of existing instruments, provides a possible solution.

  • The instrument should be widely evaluated to confirm the performance metrics recorded here. The instrument should be updated as new knowledge is generated regarding factors that affect the quality of

Methods

We used a computer-generated random sample of 30 (20%) of 151 systematic reviews that were used in the development of the instrument [5]. This sample contained 11 Cochrane and 19 non-Cochrane reviews, including meta-analyses and qualitative reviews. The topics of the reviews ranged across the spectrum of medicine [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36]. Two

Results

The sample of 30 reviews adequately covered a wide range of quality, albeit with some underrepresentation of poor-quality reviews. Overall quality scores on AMSTAR ranged from 3 to 10 (out of a maximum of 11) with a flat distribution between 3.5 and 10 and a mean percentage score of 49.4%. The overall quality scores on Sacks' instrument ranged from 5 to 16 (out of a maximum score of 24), with a mean percentage score of 41.6%, and for OQAQ, scores ranged from 3 to 10 (out of a maximum score of

Discussion

There has been a continued proliferation of (largely unvalidated) scales and checklists for assessing the quality of systematic reviews [44]. This causes confusion for those who use reviews in making clinical and policy decisions and who need to be able to distinguish good- from poor-quality reviews. There is a need for a reliable and valid quality assessment instrument that is easy to use. AMSTAR was developed to meet this need. Our aim was not to devise a truly original instrument, but to

References (54)

  • B. Shea et al.

    Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews

    BMC Med Res Methodol

    (2007)
  • B.J. Shea et al.

    External validation of a measurement tool to assess systematic reviews (AMSTAR)

    PLoS ONE

    (2007)
  • Anonymous

    Effects of adjuvant tamoxifen and of cytotoxic therapy on mortality in early breast cancer. An overview of 61 randomized trials among 28,896 women. Early Breast Cancer Trialists' Collaborative Group

    NEJM

    (1989)
  • L.J. Appel et al.

    Does supplementation of diet with “fish oil” reduce blood pressure

    Arch Intern Med

    (1993)
  • J.E. Buring et al.

    Randomized trials of aminoglycoside antibiotics: quantitative overview

    Rev Inf Dis

    (1988)
  • T.C. Chalmers et al.

    Evidence favouring the use of anticoagulants in the hospital phase of acute myocardial infarction

    NEJM

    (1977)
  • G.P. Clagett et al.

    Prevention of venous thromboembolism in general surgical patients. Results of meta-analysis

    Ann Surg

    (1988)
  • C. Counsell et al.

    Different patches in carotoid surgery

    Cochrane Library

    (1996)
  • S. Daya

    Comparison of FSH and HMG in IVF

    Cochrane Library

    (1996)
  • L. Duley et al.

    Anticonvulsants for pre-eclampsia

    Cochrane Library

    (1996)
  • J. Fanning et al.

    Meta-analysis of cisplatin, doxorubicin, and cyclophosphamide versus cisplatin and cyclophosphamide chemotherapy of ovarian carcinoma

    Obstet Gynecol

    (1992)
  • P.C. Gotzsche et al.

    Somatostatin vs placebo in bleeding oesophageal varices: randomised trial and meta-analysis

    BMJ

    (1995)
  • P. Graves

    Malaria vaccines

    Cochrane Library

    (1996)
  • W.G. Henderson et al.

    Antiplatelet or anticoagulant therapy, after coronary artery bypass surgery: a meta-analysis of clinical trials

    Ann Intern Med

    (1989)
  • E.D. Hodnett

    Alternative versus conventional delivery settings

    Cochrane Library

    (1996)
  • G.J. Hofmeyr

    Abdominal decompression

    Cochrane Library

    (1996)
  • W. Hopfenmuller

    Nackweis der therapeutischen Wirksamkeit eines Ginkgo biloba-Spezial extrakes: Meta-Analyse von 11 klinischen Studien bei Patienten mit Hirnleistungsstorungen im Alter

    Arzneimittel-Forschung

    (1994)
  • Cited by (1395)

    • Umbrella Review: Atlas of the Meta-Analytical Evidence of Early-Onset Psychosis

      2024, Journal of the American Academy of Child and Adolescent Psychiatry
    View all citing articles on Scopus
    View full text