Skip to main content
SpringerLink
Log in

A Strategy to Identify Critical Appraisal Criteria for Primary Mixed-Method Studies

  • Published:
Quality and Quantity Aims and scope Submit manuscript

Abstract

The practice of mixed-methods research has increased considerably over the last 10 years. While these studies have been criticized for violating quantitative and qualitative paradigmatic assumptions, the methodological quality of mixed-method studies has not been addressed. The purpose of this paper is to identify criteria to critically appraise the quality of mixed-method studies in the health literature. Criteria for critically appraising quantitative and qualitative studies were generated from a review of the literature. These criteria were organized according to a cross-paradigm framework. We recommend that these criteria be applied to a sample of mixed-method studies which are judged to be exemplary. With the consultation of critical appraisal experts and experienced qualitative, quantitative, and mixed-method researchers, further efforts are required to revise and prioritize the criteria according to importance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Altheide, D. L. & Johnson, J. M. (1994). Criteria for assessing interpretive validity in qualitative research. In: N. K. Denzin & Y. S. Lincoln (eds.), Handbook of Qualitative Research.Thousand Oaks, CA: Sage Publications, pp. 485–499.

    Google Scholar 

  • Berger, P. L. & Luckmann, T. (1996). The Social Construction of Reality: A Treatise in the Sociology of Knowledge.Garden City, NY: Doubleday.

    Google Scholar 

  • Berk, R. A. (1979). The construction of rating instruments for faculty evaluation: A review of the methodological issues. J. High Educ. 50(5): 650–670.

    Google Scholar 

  • Burns, N. (1989). Standards for qualitative research. Nurs.Sci.Q.2(1): 44–52.

    Google Scholar 

  • Caracelli, V. J. & Greene, J. C. (1993). Data analysis strategies for mixed-method evaluation designs. Educ. Eval. Policy. An. 15: 195–207.

    Google Scholar 

  • Caracelli, V. J. & Riggin, L. J. C. (1994). Mixed method evaluation: Developing quality criteria through concept mapping. Eval. Pract. 15: 139–152.

    Google Scholar 

  • Casebeer, A. L. & Verhoef, M. J. (1997). Combining qualitative and quantitative research methods: Considering the possibilities for enhancing the study of chronic diseases. Chronic Dis. Can. 18: 130–135.

    Google Scholar 

  • Cobb, A. K. & Hagemaster, J. N. (1987). Ten criteria for evaluating qualitative research proposals. J. Nurs. Educ. 26(4): 138–143.

    Google Scholar 

  • Creswell, J. (1998). Qualitative Inquiry and Research Design: Choosing Among Five Traditions.Thousand oaks, CA: Sage Publications.

    Google Scholar 

  • Datta, L. (1997). Multimethod evaluations: Using case studies together with other methods. In: E. Chelimsky & W. R. Shadish (eds), Evaluation for the 21st Century: A Handbook.Thousand Oaks, CA: Sage Publications, pp. 344–359.

    Google Scholar 

  • Denzin, N. K. & Lincoln, Y. S. (1994). Introduction: Entering the field of qualitative research. In: N. K. Denzin & Y. S. Lincoln (eds), Handbook of Qualitative Research.Thousand Oaks, CA: Sage Publications, pp. 1–17.

    Google Scholar 

  • Department of Clinical Epidemiology and Biostatistics, McMaster University Health Sciences Centre (1981a). How to read clinical journals: II. To learn about a diagnostic test. CMAJ 124: 703–710.

    Google Scholar 

  • Department of Clinical Epidemiology and Biostatistics, McMaster University Health Sciences Centre (1981b). How to read clinical journals: III To learn the clinical course and prognosis of disease. CMAJ 124: 869–872.

    Google Scholar 

  • Department of Clinical Epidemiology and Biostatistics, McMaster University Health Sciences Centre (1981c). How to read clinical journals: V: To distinguish useful from useless or even harmful therapy. CMAJ 124: 1156–1162.

    Google Scholar 

  • Devers, K. J. (1999). How will we know 'good' qualitative research when we see it? Beginning the dialogue in health services research. Health Serv. Res. 45(5 Part II): 1153–1188.

    Google Scholar 

  • Droitcour, J. A. (1997). Cross design synthesis: Concept and application. In: E. Chelimsky & W. R. Shadish (eds), Evaluation for the 21st Century: A Handbook.Thousand Oaks, CA: Sage Publications, pp. 360–372.

    Google Scholar 

  • Dunn, E. V. (1991). Basic standards for analytic studies in primary care research. In: P. G. Norton, M. Stewart, F. Tudiver, M. J. Bass & E. V. Dunn (eds), Primary Care Research. Newbury Park CA: Sage Publications, pp. 78–96.

    Google Scholar 

  • Elliott, R., Fischer, C. T. & Rennie, D. L. (1999). Evolving guidelines for publication of qualitative research studies in psychology and related fields. Br.J.Clin.Psychol.38(3): 215–229.

    Google Scholar 

  • Ford-Gilboe, M, Campbell, J. & Berman, H. (1995). Stories and numbers: Coexistence without compromise. Adv. Nurs. Sci. 13(1): 14–26.

    Google Scholar 

  • Forchuk, C. & Roberts, J. (1993). How to critique qualitative research articles. Can. J. Nurs. Res. 25(4): 47–56.

    Google Scholar 

  • Foster, R. L. (1997). Addressing epistemological and practical issues in multimethod research: A procedure for conceptual triangulation. Adv. Nurs. Sci. 20(2): 1–12.

    Google Scholar 

  • Fowkes, F. G. R. & Fulton, P. M. (1991). Critical appraisal of published research: Introductory guidelines. BMJ 302: 1136–1140.

    Google Scholar 

  • Gardner, M. J., Machin, D. & Campbell, M. J. (1986). Use of check lists in assessing the statistical content of medical studies. BMJ 292: 810–812.

    Google Scholar 

  • Garratt, D. & Hodkinson, P. (1998). Can there be criteria for selecting research criteria?-A hermeneutical analysis of an inescapable dilemma. Qualitative Inquiry 4(4): 515–539.

    Google Scholar 

  • Greene, J. C. & Caracelli, V. J. (eds) (1997). Advances in mixed-method evaluation: The challenges and benefits of integrating diverse paradigms. New Directions for Program Evaluation.San Francisco: Jossey-Boss Publishers.

  • Greenhalgh, T. (1997a). Assessing the methodological quality of published papers. BMJ 315(7103): 305–308.

    Google Scholar 

  • Greenhalgh, T. (1997b). Statistics for the nonstatistician. II: 'Significant' relations and their pitfalls. BMJ 315(7105): 422–425.

    Google Scholar 

  • Greenhalgh, T. & Taylor, R. (1997). Papers that go beyond numbers (qualitative research). BMJ 315: 740–743.

    Google Scholar 

  • Guba, E G. (1990), The alternative paradigm dialog. In: E. G. Guba (ed.), The Paradigm Dialog.Newbury Park, CA: Sage Publications, pp. 17–30.

    Google Scholar 

  • Guba, E. G. & Lincoln, Y. S. (1994a). Competing paradigms in qualitative research. In: N. K. Denzin & Y. S. Lincoln (eds). Handbook of Qualitative Research.Thousand Oaks, CA: Sage Publications, p. 110.

    Google Scholar 

  • Guba, E. G. & Lincoln, Y. S. (1994b). Competing paradigms in qualitative research. In: N. K. Denzin & Y. S. Lincoln (eds), Handbook of Qualitative Research.Thousand Oaks, CA: Sage Publications, pp. 105–117.

    Google Scholar 

  • Guba, E. G. & Lincoln, Y. S. (1994c). Competing paradigms in qualitative research. In: N. K. Denzin & Y. S. Lincoln (eds), Handbook of Qualitative Research.Thousand Oaks, CA: Sage Publications.

    Google Scholar 

  • Guyatt, G. H., Sackett, D. L., Cook, D.J., for the Evidence-Based Working Group. (1994). Users' guide to the medical literature. II: How to use an article about therapy or prevention, B. What were the results and will they help me in caring for my patients? JAMA 271(1): 59–63.

    Google Scholar 

  • Haughey, B. P. (1994a). Evaluating quantitative research designs: Part 1. Crit. Care Nurse 14(5): 100–102.

    Google Scholar 

  • Haughey, B. P. (1994b). Evaluating quantitative research designs: Part 2. Crit. Care Nurse 14(6): 69–72.

    Google Scholar 

  • Health Services Research (1999). 34(5).

  • Health Educational Quarterly (1992). 19(1).

  • House, E. R. (1994). Integrating the qualitative and quantitative. In: C. S. Reichardt & S. F. Rallis (eds), The Qualitative-Quantitative Debate: New Perspectives.San Francisco: Jossey-Boss Publishers.

    Google Scholar 

  • Inui, T. S. & Frankel, R. M. (1991). Evaluating the quality of qualitative research. J. Gen. Internal. Med.6: 485–486.

    Google Scholar 

  • Koch, T. (1994). Establishing rigour in qualitative research: The decision trail. J. Adv. Nurs. 19: 976–986.

    Google Scholar 

  • Kuzel, A. J. & Like, R. C.(1991). Standards of trustworthiness for qualitative studies in primary care. In: P. G. Norton, M. Steward, F. Tudiver, M. J. Bass & E. V. Dunn (eds), Primary Care Research. Newbury Park, CA: Sage Publications; pp. 138–158.

    Google Scholar 

  • Laupacis. A., Wells, G., Richardson, S. & Tugwell, P., for the Evidence-Based Medicine Working Group (1994). Users' guides to the medical literature: V. How to use an article about prognosis. JAMA 272(3): 234–237.

    Google Scholar 

  • Levine, M., Walter, S., Lee, H., Haines, T., Holbrook, A., Moyer, V., for the Evidence-Based Working Group. (1994). Users' guide to the medical literature. IV. How to use an article about harm. JAMA 271(20): 1615–1619.

    Google Scholar 

  • Lincoln, Y.S. & E. G. Guba. (1985). Naturalistic Inquiry.Newbury Park, CA: Sage Publications.

    Google Scholar 

  • Lincoln, Y. S. & Guba, E. G. (1986). But is it rigorous? Trustworthiness and authenticity in natural-istic evaluation. In: D. D. Williams (ed.), Naturalistic Evaluation. New Directions for Program Evaluation, No. 90.San Francisco: Jossey-Boss, pp. 78–84.

    Google Scholar 

  • Marshall C. (1990). Goodness criteria: Are they objective or judgement calls? In: E. G. Guba (ed.), The Paradigm Dialog.Newbury Park, CA: Sage Publications, pp. 188–197.

    Google Scholar 

  • Morgan, D. L. (1998). Practical strategies for combining qualitative and quantitative methods: Applications to health research. Qual. Health Res. 8: 362–376.

    Google Scholar 

  • Morse, J. M. (1991). Evaluating qualitative research. Qual. Health Res. 1(3): 283–286.

    Google Scholar 

  • Morse, J. M. (1986). Quantitative and qualitative research: Issues in sampling. In: P. L. Chin (ed.), Nursing Research Methodology: Issues and Implementation.Rockville, MA: Aspen Publishers.

    Google Scholar 

  • New Directions for Evaluation: Advances in Mixed-Method Evaluation: The Challenges and Benefits of Integrating Diverse Paradigms 1997; p. 74.

  • Oxman, A.D., Sackett, D. L. & Guyatt, G. H. (1993). For the Evidence-Based Medicine Working Group. Users' guides to the medical literature: 1. How to get started. JAMA 270(17): 2093–2095.

    Google Scholar 

  • Patton, M. Q. (1990). Enhancing the quality and credibility of qualitative analysis. In: Qualitative Evaluation and Research Methods, 2nd edn. Newbury Park, CA: Sage Publications.

    Google Scholar 

  • Patton, M. Q. (1999). Enhancing the quality and credibility of qualitative analysis. Health Serv. Res. 34(5 Part II): 1189–1208.

    Google Scholar 

  • Popay, J., Rogers, A. & Williams, G. (1998). Rationale and standards for the systematic review of qualitative literature in health services research. Qual. Health Res. 8(3): 341–351.

    Google Scholar 

  • Popper, K. R. (1959). The Logic of Scientific Discovery.New York: Basic Books

    Google Scholar 

  • Reichardt, C. S. & Rallis, S.F. (1994). Qualitative and quantitative inquiries are not incompatible: A call for a new partnership. In: C. S. Reichardt & S. F. Rallis (eds), The Qualitative-Quantitative Debate: New Perspectives.San Francisco: Jossey-Boss Publishers, pp. 85–92.

    Google Scholar 

  • Reid. A. (1996). What we want: Qualitative research. Can. Fam. Physician 42: 387–389.

    Google Scholar 

  • Sale, J. E. M, Lohfeld, L. & Brazil, K. (2002). Revisiting the quantitative-qualitative debate: Implications for mixed-methods research. Quality & Quantity 36: 43–53.

    Google Scholar 

  • Secker, J., Wimbush, E., Watson, J. & Milburn, K. (1995). Qualitative methods in health promotion research: Some criteria for quality. Health Educ. J. 54: 74–87.

    Google Scholar 

  • Smith, J. K. (1983). Quantitative versus qualitative research: An attempt to clarify the issue. Educational Researcher 12: 6–13.

    Google Scholar 

  • Smith, J. K. (1990). Alternative research paradigms and the problem of criteria. In: E. G. Guba (ed.), The Paradigm Diaolog.Newbury Park, CA: Sage Publications, pp. 167–187.

    Google Scholar 

  • Sparkes, A. C. (2001). Myth 94: Qualitative health researchers will agree about validity. Qual. Health Res. 11(4): 538–552.

    Google Scholar 

  • Tashakkori, A. & Teddlie, C. (1998). Mixed Methodology: Combining Qualitative and Quantitative Approaches.Thousand Oaks, CA: Sage Publications.

    Google Scholar 

  • Wilson, H. S. (1985). Qualitative studies: From observations to explanations. Journal Nurs. Adm. 8–10.

  • Yin, R. K. (1999). Enhancing the quality of case studies in health services research. Health Serv. Res. 34(5 Part II): 1209–1224.

    Google Scholar 

  • Yonge, O. & Slewin, L. (1988). Reliability and validity: Misnomers for qualitative research. Can. J Nurs. Res.20(2): 61–67.

    Google Scholar 

Download references

Authors
  1. Joanna E. M. Sale
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kevin Brazil
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Sale, J.E.M., Brazil, K. A Strategy to Identify Critical Appraisal Criteria for Primary Mixed-Method Studies. Quality & Quantity 38, 351–365 (2004). https://doi.org/10.1023/B:QUQU.0000043126.25329.85

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/B:QUQU.0000043126.25329.85

Search

Navigation