Developing a program theory of patient engagement in patient-oriented research and the impacts on the health care system: protocol for a rapid realist review
=============================================================================================================================================================

* Elaine Zibrowski
* Shelagh McDonald
* Heather Thiessen
* Ray VanDusen
* Catherine Boden
* Tracey Carr
* Donna Goodridge
* Charlene Haver
* Darcy Marciniuk
* Christine Stobart
* Tanya Verrall
* Gary Groot

## Abstract

**Background:** The patient-oriented research (POR) discourse has been criticized as being fragmented, lacking consistent terminology and having few evaluative studies. Our research team will use rapid realist review methodology to generate broad, process-based program theory regarding how partnering patients with researchers in POR generates an impact within a health care system.

**Methods:** This protocol for a rapid realist review will involve multiple steps, including research question development; preliminary program theory and search strategy development; study selection and appraisal; data extraction, analysis and synthesis; and program theory refinement. We will be guided by the Realist and Meta-narrative Evidence Syntheses: Evolving Standards (RAMESES) publication standards for realist synthesis. Unlike traditional reviews, a realist review aims to discover and understand causal processes that exist within a complex environment, asking questions regarding what works for whom, under what circumstances, how and why. Our multidisciplinary team consists of patient partners, health care professionals, a health sciences librarian and health services researchers. Patient partners are full research partners, supporting development of our guiding research question and identifying community partners and stakeholder groups to disseminate our findings. Patient partners will be asked to recommend literature sources, to review and vet our set of search terms, and to review, evaluate and reflect on our initial program theory in light of their personal, lived expertise.

**Interpretation** We will share the results of our rapid realist review with community partners and stakeholder groups. We will also disseminate our program theory by means of publication in a peer-reviewed journal and presentation at scientific conferences.

**Plain language summary**: We are a team of patient partners, health care providers, a health sciences librarian, health services researchers and professionals from a provincial research support centre in Saskatoon, Canada. As a team, we are trying to understand better how involving patients, family members, caregivers and other members of the public in research may lead to improving one or more aspects of health care systems. This protocol will guide the development of a rapid realist review, to locate existing patient-oriented research studies that have been published in scientific journals or have been communicated by government-sponsored initiatives in Canada and internationally. Realist reviews focus on interpretation, critique and deepening of understanding regarding complex processes with the development of theory. From our set of located studies, we aim to develop a theory answering the following research question: “How, why, to what extent and in what contexts does patient-oriented research produce impacts on individual collaborators, research processes, communities and health care systems?” We plan to share what we learn about patient-oriented research with other community partners, stakeholders, academic researchers and health care professionals.

Patient-oriented research (POR) intervenes on the traditional research role of patients, family, caregivers and other members of the general public.1 Patient-oriented research recognizes that the personal health journeys of individuals enable them to develop lived expertise regarding health care systems. This lived expertise makes patients valuable members of a research team.1 Patient-oriented research anticipates that partnering patients with researchers will enhance research quality and will “improve health care policies and practices across the system, ultimately improving health outcomes.”2

The POR discourse has been described as being in a catch-22 paradox.3 Information from reviews and prospective studies have highlighted POR’s potential to improve research conduct, and diverse impacts have been reported for individual collaborators, health services, patient care delivery and communities.4–12 However, POR has been repeatedly criticized for being fragmented, lacking consistent terminology and having few evaluative studies.3,13,14 Consequently, researchers have advocated that if POR is to reach its potential, rigorous measurement frameworks that can provide better-quality evidence of its impacts are needed.3,5,13,14

To advance these metrics, we believe that we must first have a better fundamental and theoretical understanding of how impacts unfold within POR studies themselves. Although theory has been underrecognized in research related to health care improvement, it can afford examination of research problems under different perspectives15,16 These lenses can, in turn, allow stakeholders to view and understand research findings across a wider pane of importance and applicability, and uncover processes and their generating influences that may not have been immediately visible.17

Our study objective is to develop a program theory for POR and its impacts on the health care system. Our primary research question is “How, why, to what extent and in what contexts does POR produce impacts on individual collaborators, research processes, communities and health care systems?” Our associated research questions are as follows: What contexts support or hinder POR to enable an impact; and within these contexts, what mechanisms are at work to generate a given impact?

## Methods

### Study design

This is a protocol for a rapid realist review. Our rapid realist review will be guided by the Realist and Meta-narrative Evidence Syntheses: Evolving Standards (RAMESES) publication guideline for quality reporting of a realist review.18 We will use a 5 step process which includes: 1) research question development, 2) preliminary program theory and search strategy development, 3) study selection and appraisal, 4) data extraction, analysis and synthesis and 5) program theory refinement. We aim to complete our review within 1 year.

Originally developed by Pawson, a realist review is a type of narrative review that is grounded in the philosophical tradition of realism,19–24 and is focused on interpretation, critique and deepening of understanding.24 Realism asserts that both the material and social worlds are “real,” and these can contribute to an effect or change. All interventions or programs possess “theories incarnate,” and therefore, whenever an intervention or program is undertaken, the intervention or program is testing a theory regarding what, how and when it may lead to change.19–23,25 This theory may not be explicit, and a realist review seeks to illuminate embedded theories through the development of generative, causal hypotheses regarding how, for whom and in what contexts an intervention or program may “work.”19–23,25

Realist reviews embrace complexity and therefore can be useful when exploring phenomena that occur across varied contexts and may be associated with inconsistent outcomes.26,27 Given that health care is enacted within complex systems28 and the evidence regarding POR impacts appears fragmented,3,13,14 we selected realist review as our methodological approach.

Realist reviews can follow a traditional or rapid approach. Although both types involve the same research activities, a rapid realist review emphasizes the value of involving a local reference group, including experts, as its stakeholders.29 Stakeholders include individuals who are knowledge users and client representatives. Rapid realist reviews examine nascent issues and seek to produce knowledge that can be useful to policy-making. To be responsive to emerging issues, a rapid realist review typically evolves over a period of 1 year or less.29 We selected rapid realist review because of its emphasis on examining an emergent issue, seeking a review product that holds potential value for policy-makers, and its recognition of what diverse stakeholders, including lay members, can bring to theory development.

A realist review produces a program theory derived from context-mechanism-outcome configurations (commonly referred to as CMO).18–22,24 Realist research uses retroduction, a form of scientific reasoning that “refers to the identification of hidden causal forces that lie behind identified patterns or changes in those patterns.”30 Context refers to physical and social conditions in which an intervention has been under-taken. Mechanisms can be explicit or hidden and will be triggered only under particular contextual conditions.18–20,30,31 Outcomes are triggered by mechanisms, can be explicit or hidden and involve intended or unintended consequences.31,32

### Stakeholders

Our stakeholders include 4 experienced realist researchers (a clinician–scientist and 3 health services researchers), a clinician–scientist who is a policy-maker, a health sciences librarian, a postdoctoral fellow and 2 doctoral trained research specialists who work at a POR support centre located in Saskatoon, Saskatchewan. The Saskatoon POR centre supported in the recruitment of 3 individuals who had lived experience with a health care system (locally referred to as patient partners) for our stakeholder team, by hosting a networking event, soliciting referrals from colleagues and emailing advertisements to the centre’s patient partners who had self-identified research interests. Our patient partners have extensive lived experience with a chronic health condition or have cared for a family member with a chronic health condition.

Several stakeholders are representatives of health care–related advisory boards and committees at local, provincial, national and international levels.

### Research question development

Our review question was developed in 2 steps with patient partners. During a single meeting held on Jan. 24, 2019, our stakeholders divided themselves into small groups, each with a patient partner and 2 other stakeholders. The groups prioritized POR topics they would be most interested in exploring with a rapid realist review. Our entire team identified impacts of POR as a high-priority topic. On Jan. 30, 2019, a team member presented our prioritized topics during a realist research seminar at the University of Saskatchewan (Saskatoon, Sask.). Seminar attendees reviewed our ideas and supported us to reframe them from a realist perspective. Our research question was sent to the entire stakeholder team for their refinement and approval.

### Preliminary program theory and search strategy

We will develop a preliminary program theory to support our search strategy. In a realist review, this theory is a first, rough attempt to illuminate the phenomenon under examination.18,25 We will use input from our stakeholders and conduct a search of POR literature to identify potential contexts, mechanisms and existing theories that may support our work.

We have conceptualized POR with the broad definition proposed by the Canadian Institutes of Health Research (CIHR).1,2 Patient-oriented research is informed by patient priorities and seeks to improve patient care or an aspect of a health care system. Patient engagement is a central tenant of POR. We have operationalized the concept of a health care system according to the definition provided by the World Health Organization.33 This definition extends the concept of health care beyond services solely delivered by formal medical organizations. Spouses, family members and other community members are recognized as important providers of care.34 We will be guided by the framework of Aubin and colleagues,13 which conceptualizes POR impacts consistent with realist definitions of proximal, intermediate and final level impacts.31,32

We will seek documents that provide the most valuable information and afford us to explore and interrogate aspects of our initial program theory. The librarian will work with another team member to develop a set of search terms to locate published literature within 2 bibliographic databases (CINAHL and Ovid MEDLINE) and 1 abstract and citation database (Scopus).

We will first ask our team members to review the generated list and identify published articles they perceive as the most relevant to our review. The full-text version of recommended articles will be examined for words, phrases or author-derived keywords that may be useful to include as search terms. We will develop and refine the set of search terms over several iterations, with reviews and vetting of search terms by our stakeholders along with pilot searches in the 3 databases.

We will develop and refine a preliminary search strategy in MEDLINE, and then optimize it for CINAHL and Scopus. No date or study type limits will be applied. We also anticipate that we will locate additional published sources by reviewing the references cited within recommended articles (pearling) and reviewing the table of contents of selected journals (hand searching). Systematic review software will be used to de-duplicate and collate bibliographic records into a single data file.

Realist methodology acknowledges that grey literature sources, such as government documents, evaluation reports and information disseminated by websites, can provide valid information toward a realist review. Similar to our search for published articles, we anticipate that locating grey literature will require an iterative process involving recommendations from our stakeholders and reviewing the public websites of key international POR initiatives from Canada (Strategy for Patient-Oriented Research [SPOR] led by CIHR),1 United Kingdom (National Standards for Public Involvement led by the National Institute for Health Research)35 and the United States (Patient-Centered Outcomes Research Institute).36

### Study selection and appraisal

Two team members will independently screen published and grey literature. A source will be included if it satisfies all of our inclusion criteria (Box 1). As POR embraces the value of seeking collaboration with affected individuals and communities, in a similar manner as participatory action and community-based research,37 studies that follow these models will be eligible for inclusion. We will look for studies across a wide spectrum of health care including those related to public health, as authors have recently advocated that studies related to health promotion, protection and prevention may be underrepresented by current POR definitions.38 Studies or projects involving quantitative designs, qualitative approaches or mixed methods will be eligible for inclusion. We will use a random sample of 10% of the articles that we locate to pilot our eligibility criteria and train the raters to use them. Full-text versions will be located for all studies and projects that meet the set of inclusion criteria.

Box 1: 
### **Inclusion and exclusion criteria**

**Inclusion criteria**

*   The document is written in English

*   The document was accompanied by an abstract, executive summary or synopsis that disseminated the purpose, methodology and findings of the study or project

*   The objective of the research study or project was informed by patient priorities and sought to improve patient care or an aspect of a health care system

*   The research team that conducted the research study or project must have included both decision-makers (for example, health care practitioners, policy-makers, managers of local health authority) and at least 1 patient, caregiver, or family member, or another member of the general public (patient partner)

*   The patient partners must have actively participated in the work involved in the governance, priority-setting, research or knowledge translation of a research study or project

**Exclusion criteria**

*   The document is not written in English

*   The document was not accompanied by an abstract, executive summary or synopsis that disseminated the purpose, methodology and findings of the study or project

*   The document is an editorial, letter to the editor, commentary or study protocol that did not present any data

*   The involvement of a patient partner in the study or project was limited to their contribution of data as a study or project participant

Realist research appraises evidence according to Pawson’s criteria of rigour and relevance under a “fitness to purpose” lens.20–23 Rigour relates to the methodological quality of a study, while relevance refers to the extent that a research study can potentially contribute information toward the development of program theory.20–23 Based on the full-text descriptions of a study’s purpose, methods and analysis and the claims made by its authors, 2 members of our team will decide if that study is a good fit for our review. A study of good fit is one that appears to be credible and trustworthy, and contains descriptive information that can potentially support our team in developing a theory regarding how POR affects a health care system.20–23 To render a majority decision, if the 2 raters cannot agree whether a study is a good fit, a third team member will be asked to appraise it.

### Patient engagement

Patient partners are full research partners. The 3 patient partners have supported the development of our guiding research question and identified community partners and stakeholders to disseminate our findings. As we conduct our review, we will ask them to recommend literature, and review our set of search terms. They will attend our stakeholder meetings, and will be asked to review, evaluate and reflect on our initial program theory in light of their personal, lived expertise. They will support the translation of our findings to academic and general public audiences. Patient partners will be offered honoraria, and their travel expenses will be reimbursed. Honoraria and travel will be funded consistent to provincial guidelines.

### Data analysis

Two team members, with support from the experienced realist researchers in our group, will extract and synthesize data from the studies that were appraised as appropriate for our review. From the full-text version of each study, information will be extracted regarding its setting and participants, study or project design and the activities that participants were involved in, along with other relevant information reported by the original authors.23,39 These data will be recorded in a spreadsheet.

Consistent with retroduction, which hallmarks a realist review, explanatory accounts will then be extracted. These will include causal statements that identify enabling or constraining factors that were present in an intervention or study, the impact of those factors on 1 or more mechanisms and the outcomes that were produced.23,40,41 We will express explanatory accounts in the form of “if-then” propositional statements that assume an “x-y because of z” relationship. That is, a given outcome “y” occurs because mechanism “z” fires when the contextual condition “x” is present.32 Once the entire set of explanatory accounts has been developed, we will consolidate these, and then look for connections and patterns occurring within and between each explanatory account.41,42 In a realist review, these patterns are referred to as demi-regularities.31,32 We will synthesize demi-regularities into thematic groups. We will then develop a CMO configuration for each thematic group.

The CMO configurations will support us to develop and refine an initial program theory. This theory may involve elements of our first, rough theory that we used at the onset of our rapid realist review.

#### Program theory refinement

The final stage of a rapid realist review involves the presentation and testing of the initial program theory with stakeholders. 20–23,31 During in-person meetings, we will ask our team to review and evaluate our initial program theory. Patient partners will be asked to reflect on the program theory in light of their personal POR experiences. Our team may confirm, refute, identify gaps or suggest refinements to the theory. During our discussions, we may revisit full-text versions of studies that were included in our rapid realist review or our data extractions. We may also opt to conduct additional searches of academic or grey literature. It is through this iterative process of stakeholder review and refinement that an initial program theory is tested and validated in a realist review.20–23,31,34 We will seek ongoing refinements to our theory to the point where we suspect that additional re-examination of our CMO configurations is unlikely to result in any other further modifications. At this point, we will sense theoretical sufficiency or saturation.19–21 Our final, refined program theory will be of the middle range.17,19–21,30,39 Middle-range theories are testable but not restricted to a particular context or study case.20–22,31,32

### Ethics approval

As our research will involve a secondary review of previously disseminated information, we were granted a formal exemption from institutional review by the Behavioural Research Ethics Board at the University of Saskatchewan.

## Interpretation

We will prepare a summary report that we anticipate will be disseminated to the national SPOR units. We also anticipate delivering presentations about our review at national SPOR summits and to patient advisory committees organized locally and provincially. Patient partners will provide guidance on how to communicate our findings with a genre that can reach diverse, general public audiences.

In terms of academic platforms, we anticipate disseminating our review in a peer-reviewed journal, and through presentations at scientific conferences. Patient partners will be invited to attend and co-present our findings at these conferences. We do not anticipate that any ethical issues will occur during our review.

### Limitations

As with any other review article, our research cannot comprehensively consider the entire POR discourse. Our review is limited to English-disseminated information sources. Realist reviews are not intended to identify pathways to achieving all possible outcomes, nor can this methodology exhaustively explain all mechanisms that are capable of generating changes.

### Conclusion

We will develop a program theory to illuminate how POR produces health care system impacts, for whom, under which circumstances and why. We anticipate that our research will be of interest to patients and their family members, caregivers, researchers, health care professionals and policy-makers.

## Footnotes

*   **Competing interests:** None declared.

*   This article has been peer reviewed.

*   **Contributors:** Elaine Zibrowski contributed to conceptualization, funding acquisition, methodology, project administration, writing the original draft and reviewing and editing the final version. Shelagh McDonald, Heather Thiessen and Ray vanDusen contributed to conceptualization and writing-review of the final version. Catherine Boden, Donna Goodridge and Tanya Verrall contributed to conceptualization, methodology and writing-review of the final version. Tracey Carr contributed to conceptualization, methodology, project administration, supervision, writing-review and editing the final version. Charlene Haver and Darcy Marciniuk contributed to conceptualization and writing-review of the final version. Christine Stobart contributed to conceptualization, funding acquisition, project administration and writing-review of the final version. Gary Groot contributed to conceptualization, funding acquisition, methodology, project administration, supervision and writing-review of the final version. All authors revised the article critically for important intellectual content, approved the final version to be published and agreed to be accountable for all aspects of the work.

*   **Funding:** This rapid realist review is supported with in-kind funds from the Saskatchewan Centre for Patient-Oriented Research and the College of Medicine at the University of Saskatchewan.

*   **Data sharing:** No data are associated with this protocol. It is anticipated that data generated from the rapid realist review will be made available in response to a reasonable request sent to Dr. Zibrowski.

*   **Supplemental information:** For reviewer comments and the original submission of this manuscript, please see [www.cmajopen.ca/content/8/3/E530/suppl/DC1](http://www.cmajopen.ca/content/8/3/E530/suppl/DC1).

## References

1.  (2011) Canada’s strategy for patient-oriented research: improving health outcomes through evidence-informed care (Canadian Institutes of Health Research, Ottawa) modified 2012 Jan 24. Available: [https://cihr-irsc.gc.ca/e/44000.html](https://cihr-irsc.gc.ca/e/44000.html). accessed 2020 Feb. 3.
    
    

2.  Strategy for patient-oriented research (Canadian Institutes of Health Research, Ottawa) modified 2019 June 27. Available: [www.cihr-irsc.gc.ca/e/41204.html](http://www.cihr-irsc.gc.ca/e/41204.html). accessed 2019 June 30.
    
    

3.  Manafo E, Petermann L, Mason-Lai P, et al. (2018) Patient engagement in Canada: a scoping review of the ‘how’ and ‘what’ of patient engagement in health research [published erratum in *Health Res Policy Syst* 2018;16:24]. Health Res Policy Syst 16:5.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmajo%2F8%2F3%2FE530.atom) 

4.  Mockford C, Staniszewska S, Griffiths F, et al. (2012) The impact of patient and public involvement on UK NHS health care: a systematic review. Int J Qual Health Care 24:28–38.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1093/intqhc/mzr066&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=22109631&link_type=MED&atom=%2Fcmajo%2F8%2F3%2FE530.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000299096200005&link_type=ISI) 

5.  Brett J, Staniszewska S, Mockford C, et al. (2014) Mapping the impact of patient and public involvement on health and social care research: a systematic review. Health Expect 17:637–50.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1111/j.1369-7625.2012.00795.x&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=22809132&link_type=MED&atom=%2Fcmajo%2F8%2F3%2FE530.atom) 

6.  Brett J, Staniszewska S, Mockford C, et al. (2014) A systematic review of the impact of patient and public involvement on service users, researchers and communities. Patient 7:387–95.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1007/s40271-014-0065-0&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=25034612&link_type=MED&atom=%2Fcmajo%2F8%2F3%2FE530.atom) 

7.  Lechelt LA, Rieger JM, Cowan K, et al., Alberta Head and Neck Cancer Priority Setting Partnership Steering Committee (2018) Top 10 research priorities in head and neck cancer: results of an Alberta priority setting partnership of patients, caregivers, family members, and clinicians. Head Neck 40:544–54.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmajo%2F8%2F3%2FE530.atom) 

8.  Breault LJ, Rittenbach K, Hartle K, et al. (2018) The top research questions asked by people with lived depression experience in Alberta: a survey. CMAJ Open 6:E398–405.
    
    [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToiY21ham8iO3M6NToicmVzaWQiO3M6ODoiNi8zL0UzOTgiO3M6NDoiYXRvbSI7czoyMDoiL2NtYWpvLzgvMy9FNTMwLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

9.  Crocker JC, Ricci-Cabello I, Parker A, et al. (2018) Impact of patient and public involvement on enrolment and retention in clinical trials: systematic review and meta-analysis. BMJ 363:k4738.
    
    [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE4OiIzNjMvbm92MjhfMTAvazQ3MzgiO3M6NDoiYXRvbSI7czoyMDoiL2NtYWpvLzgvMy9FNTMwLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

10. Khan A, Spector ND, Baird JD, et al. (2018) Patient safety after implementation of a coproduced family centered communication programme: multicenter before and after intervention study. BMJ 363:k4764.
    
    [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNjMvZGVjMDRfOC9rNDc2NCI7czo0OiJhdG9tIjtzOjIwOiIvY21ham8vOC8zL0U1MzAuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

11. Nelson RG, Pankratz VS, Ghahate DM, et al. (2018) Home-based kidney care, patient activation, and risk factors for CKD progression in Zuni Indians: a randomized, controlled clinical trial. Clin J Am Soc Nephrol 13:1801–9.
    
    [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiY2xpbmphc24iO3M6NToicmVzaWQiO3M6MTA6IjEzLzEyLzE4MDEiO3M6NDoiYXRvbSI7czoyMDoiL2NtYWpvLzgvMy9FNTMwLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

12. Krewulak KD, Sept BG, Stelfox HT, et al. (2019) Feasibility and acceptability of family administration of delirium detection tools in the intensive care unit: a patient-oriented pilot study. CMAJ Open 7:E294–9.
    
    [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToiY21ham8iO3M6NToicmVzaWQiO3M6ODoiNy8yL0UyOTQiO3M6NDoiYXRvbSI7czoyMDoiL2NtYWpvLzgvMy9FNTMwLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

13. Aubin D, Hebert M, Eurich D (2019) The importance of measuring the impact of patient-oriented research. CMAJ 191:E860–4.
    
    [FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czoxMToiMTkxLzMxL0U4NjAiO3M6NDoiYXRvbSI7czoyMDoiL2NtYWpvLzgvMy9FNTMwLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

14. Staniszewska S, Herron-Marx S, Mockford C (2008) Measuring the impact of patient and public involvement: the need for an evidence base. Int J Qual Health Care 20:373–4.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1093/intqhc/mzn044&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=18836184&link_type=MED&atom=%2Fcmajo%2F8%2F3%2FE530.atom) 

15. Davidoff F, Dixon-Woods M, Leviton L, et al. (2015) Demystifying theory and its use in improvement. BMJ Qual Saf 24:228–38.
    
    [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjI0LzMvMjI4IjtzOjQ6ImF0b20iO3M6MjA6Ii9jbWFqby84LzMvRTUzMC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

16. Bonell C, Fletcher A, Morton M, et al. (2012) Realist randomised controlled trials: a new approach to evaluating complex public health interventions. Soc Sci Med 75:2299–306.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/j.socscimed.2012.08.032&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=22989491&link_type=MED&atom=%2Fcmajo%2F8%2F3%2FE530.atom) 

17. Reeves S, Albert M, Kuper A, et al. (2008) Why use theories in qualitative research? BMJ 337:a949.
    
    [FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE2OiIzMzcvYXVnMDdfMy9hOTQ5IjtzOjQ6ImF0b20iO3M6MjA6Ii9jbWFqby84LzMvRTUzMC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

18. Wong G, Greenhalgh T, Westhorp G, et al. (2013) RAMESES publication standards: realist syntheses. BMC Med 11:21.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1186/1741-7015-11-21&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=23360677&link_type=MED&atom=%2Fcmajo%2F8%2F3%2FE530.atom) 

19. Pawson R, Tilley N (1997) Realistic evaluation (Sage Publications Ltd. Thousand Oaks (CA)).
    
    

20. Pawson R, Greenhalgh T, Harvey G, et al. (2005) Realist review — a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy 10(Suppl 1):21–34.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1258/1355819054308530&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=16053581&link_type=MED&atom=%2Fcmajo%2F8%2F3%2FE530.atom) 

21. Pawson R (2006) Evidence-based policy: a realist perspective (Sage Publications Ltd).
    
    

22. Pawson R (2003) Assessing the quality of evidence in evidence-based policy: Why, how and when? ESRC Research Methods Programme Working Paper no. 1 (Economic & Social Research Council, Swindon (UK)).
    
    

23. Rycroft-Malone J, McCormack B, Hutchinson AM, et al. (2012) Realist synthesis: illustrating the method for implementation research. Implement Sci 7:33.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1186/1748-5908-7-33&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=22515663&link_type=MED&atom=%2Fcmajo%2F8%2F3%2FE530.atom) 

24. Greenhalgh T, Thorne S, Malterud K (2018) Time to challenge the spurious hierarchy of systematic over narrative reviews? Eur J Clin Invest 48:e12931.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1111/eci.12931&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=29578574&link_type=MED&atom=%2Fcmajo%2F8%2F3%2FE530.atom) 

25. Goodridge D, Westhorp G, Rotter T, et al. (2015) Lean and leadership practices: development of an initial realist program theory. BMC Health Serv Res 15:362.
    
    

26. Duddy C, Wong G (2018) Explaining variations in test ordering in primary care: protocol for a realist review. BMJ Open 8:e023117.
    
    [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYm1qb3BlbiI7czo1OiJyZXNpZCI7czoxMToiOC85L2UwMjMxMTciO3M6NDoiYXRvbSI7czoyMDoiL2NtYWpvLzgvMy9FNTMwLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

27. Reeves S (2015) The importance of realist synthesis for the interprofessional field. J Interprof Care 29:1–2.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=25424173&link_type=MED&atom=%2Fcmajo%2F8%2F3%2FE530.atom) 

28. Greenhalgh T, Papoutsi C (2018) Studying complexity in health services research: desperately seeking an overdue paradigm shift. BMC Med 16:95.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1186/s12916-018-1089-4&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=29921272&link_type=MED&atom=%2Fcmajo%2F8%2F3%2FE530.atom) 

29. Saul JE, Willis CD, Bitz J, et al. (2013) A time-responsive tool for informing policy making: rapid realist review. Implement Sci 8:103.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1186/1748-5908-8-103&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=24007206&link_type=MED&atom=%2Fcmajo%2F8%2F3%2FE530.atom) 

30. (2017) Retroduction in realist evaluation: the RAMESES II Project (Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford (UK)) Available: [www.ramesesproject.org/media/RAMESES\_II_Retroduction.pdf](http://www.ramesesproject.org/media/RAMESES_II_Retroduction.pdf), pp 1–3. accessed 2020 Feb. 3.
    
    

31. Jagosh J, Macaulay AC, Pluye P, et al. (2012) Uncovering the benefits of participatory research: implications of a realist review for health research and practice. Milbank Q 90:311–46.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1111/j.1468-0009.2012.00665.x&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=22709390&link_type=MED&atom=%2Fcmajo%2F8%2F3%2FE530.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000305388900005&link_type=ISI) 

32. Jagosh J, Pluye P, Wong G, et al. (2014) Critical reflections on realist review: insights from customizing the methodology to the needs of participatory research assessment. Res Synth Methods 5:131–41.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmajo%2F8%2F3%2FE530.atom) 

33. (2007) Everybody’s business: strengthening health systems to improve health outcomes — WHO’s framework for action (World Health Organization, Geneva).
    
    

34. White F (2015) Primary health care and public health: foundations of universal health systems. Med Princ Pract 24:103–16.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmajo%2F8%2F3%2FE530.atom) 

35. INVOLVE [main page] (INVOLVE, Southampton (UK)) Available: [www.invo.org.uk](http://www.invo.org.uk). accessed 2019 June 30.
    
    

36. Patient-Centred Outcomes Research Institute (PCORI) [main page] (PCORI, Washington (DC)) Available: [www.pcori.org](http://www.pcori.org). accessed 2019 June 30.
    
    

37. Bell T, Vat LE, McGavin C, et al. (2019) Co-building a patient-oriented research curriculum in Canada. Res Involv Engagem 5:7.
    
    

38. Pauly B, Urbanoski K, Hartney E, et al. (2019) What is missing from “Patient-Oriented Research?” A view from public health systems and services. Healthc Policy 15:10–9.
    
    

39. Lawal AK, Groot G, Goodridge D, et al. (2019) Development of a program theory for clinical pathways in hospitals: protocol for a realist review. Syst Rev 8:136.
    
    

40. Groot G, Waldron T, Carr T, et al. (2017) Development of a program theory for shared decision-making: a realist review protocol. Syst Rev 6:114.
    
    

41. Pearson M, Brand SL, Quinn C, et al. (2015) Using realist review to inform intervention development: methodological illustration and conceptual platform for collaborative care in offender mental health. Implement Sci 10:134.
    
    

42. Carr T, Quinlan E, Robertson S, et al. (2017) Adapting realist synthesis methodology: the case of workplace harassment interventions. Res Synth Methods 8:496–505.
    
    

*   Copyright 2020, Joule Inc. or its licensors