Research

Assessment of scalability of evidence-based innovations in community-based primary health care: a cross-sectional study

Ali Ben Charif, Kasra Hassani, Sabrina T. Wong, Hervé Tchala Vignon Zomahoun, Martin Fortin, Adriana Freitas, Alan Katz, Claire E. Kendall, Clare Liddy, Kathryn Nicholson, Bojana Petrovic, Jenny Ploeg and France Légaré
November 02, 2018 6 (4) E520-E527; DOI: https://doi.org/10.9778/cmajo.20180143

Abstract

Background: In 2013, the Canadian Institutes of Health Research funded 12 community-based primary health care research teams to develop evidence-based innovations. We aimed to explore the scalability of these innovations.

Methods: In this cross-sectional study, we invited the 12 teams to rate their evidence-based innovations for scalability. Based on a systematic review, we developed a self-administered questionnaire with 16 scalability assessment criteria grouped into 5 dimensions (theory, impact, coverage, setting and cost). Teams completed a questionnaire for each of their innovations. We analyzed the data using simple frequency counts and hierarchical cluster analysis. We calculated the mean number and standard deviation (SD) of innovations that met criteria within each dimension that included more than 1 criterion. The analysis unit was the innovation.

Results: The 11 responding teams evaluated 33 evidence-based innovations (median 3, range 1–8 per team). The innovations focused on access to care and chronic disease prevention and management, and varied from health interventions to methodological innovations. Most of the innovations were health interventions (n = 21), followed by analytical methods (n = 4), conceptual frameworks (n = 4), measures (n = 3) and strategies to build research capacity (n = 1). Most (29) met criteria in the theory dimension, followed by impact (mean 22.3 [SD 5.6] innovations per dimension), setting (mean 21.7 [SD 8.5]), cost (mean 17.5 [SD 2.1]) and coverage (mean 14.0 [SD 4.1]). On average, the innovations met 10 of the 16 criteria. Adoption was the least assessed criterion (n = 9). Most (20) of the innovations were highly ranked for scalability.

Interpretation: Scalability varied among innovations, which suggests that readiness for scale up was suboptimal for some innovations. Coverage remained largely unaddressed; further investigation of this critical dimension is necessary.

The development and implementation of evidence-based innovations have been strongly incentivized as part of primary care reform in Canada.1 The research focus has been on generating enough evidence of sufficient rigour to qualify innovations as evidence-based.2–4 However, there is a growing gap between development of evidence-based innovations as research projects and their widespread implementation as standard care.5–7 Scale up or spread is a form of knowledge translation that aims to increase the reach and adoption of innovations.8–13 Successful scale up follows several steps: 1) assessment of scalability (potential for scale up) of the innovation (e.g., effectiveness, cost-effectiveness), 2) development of a scale-up strategy, 3) strategy implementation, 4) strategy evaluation and 5) assuring sustainability.13–16 Rigorous evaluation of scale up includes measuring coverage (number of people, organizations or systems that adopt the innovation over the number targeted).2,17 In Canada, however, there are few systematic efforts to facilitate and support scale up and few theoretical, conceptual or practical frameworks for guiding the first step, scalability assessment.13,14 This leads to a lag between evidence generation and scale up, and denies patients and communities access to effective services.7,18–20

In 2013, 12 research teams were funded by the Canadian Institutes of Health Research to conduct cross-jurisdictional research in improving access to community-based primary health care for vulnerable populations, and chronic disease prevention and management.21 As a funding condition, teams were required to collaborate in sharing their findings and lessons learned, build capacity and plan for scale up.22 In the present study, we aimed to explore the scalability of the teams’ evidence-based innovations.

Methods

Study design

In this cross-sectional study, we invited the 12 research teams to rate the scalability of their evidence-based innovations. An evidence-based innovation was defined as a program, model, approach, tool, instrument, indicator, algorithm, service, idea, policy or practice whose evidence base has been established.23,24

Setting and participants

We conducted the study between August and December 2017 together with the teams. The teams covered all of Canada except for 1 province (Saskatchewan) and 1 territory (Yukon Territory).21,22 They included over 100 stakeholders in community-based primary health care.21,22 Their research was programmatic, i.e., each team worked on varied but conceptually linked projects bound together by the themes of access to care for vulnerable populations, and chronic disease prevention and management.21,22

Data collection

We created a 1-page self-administered questionnaire containing the key elements for assessing the scalability of the teams’ evidence-based innovations (Appendix 1, available at www.cmajopen.ca/content/6/4/E520/suppl/DC1). We designed the questionnaire based on the results of a systematic review of scale-up strategies in primary care2 and the recommendations of 2 scale-up guides.13,14 The teams were invited to rate the scalability of their evidence-based innovations and were asked for details about their team and the name, type and aim of the innovation. The questionnaire included 16 criteria for assessing scalability, grouped into 5 dimensions: 1) the theory used to develop the innovation, 2) impact (e.g., has efficacy been assessed), 3) likely coverage of the innovation (e.g., has adoption been assessed), 4) alignment of the innovation with the setting (e.g., is it compatible with similar interventions in the same setting) and 5) cost (e.g., is scale up affordable). For each criterion, there were 5 response options: 1) “Yes” (criterion was assessed), 2) “No” (criterion was not assessed), 3) “UE” (criterion was under evaluation), 4) “NP” (criterion assessment was not planned) and 5) “NA” (not applicable). Space was provided for additional comments on each criterion. The questionnaire was piloted by 1 of the 12 teams (S.T.W.’s team) and was revised to improve understanding of the instructions and examples of evidence-based innovations. All teams were asked to return their completed responses by email. A maximum of 3 reminder messages were sent no later than 3 weeks after the first message. A team could rate more than 1 of their innovations (1 questionnaire each).

Data analysis

Two of the authors (A.B.C. and H.T.V.Z.) independently classified the innovations into 1 of 5 mutually exclusive types according to predefined themes. Any discrepancies were resolved through consensus with the senior author (F.L.). The themes corresponded to the following types of evidence-based innovations:

  • Health intervention: an act performed for, with or on behalf of a person or a population to improve, maintain, promote or modify health functioning or health conditions (e.g., preventive strategy, screening program, decision aid)25

  • Analytical method: a generic or systematic process combining the scientific method with the use of a formal process to solve a research problem26,27

  • Conceptual framework: a set of concepts or abstractions linked and arranged rationally according to their relevance to a common theme28

  • Measure: any instrument, measure or indicator29

  • Strategy to build research capacity: a process of individual or institutional development that leads to a greater ability to perform useful research.30,31

Team respondents were involved in all steps of the study. Together, we grouped the response options for the 16 scalability assessment criteria into 3 categories: criterion met (responses corresponding to “criterion was assessed” and “criterion was under evaluation”), criterion not met (responses corresponding to “criterion was not assessed” and “criterion assessment was not planned”) and not applicable. Thus, each innovation was scored on the number of criteria met. Then we ranked the 16 criteria according to the number of innovations that met each of these criteria. For the dimensions that include more than 1 criterion, we calculated the mean number and standard deviation (SD) of innovations per dimension for which criteria were met. We used analysis of variance to compare the number of criteria met per innovation.

We analyzed the data using simple frequency counts and hierarchical cluster analysis. The unit of analysis was the innovation. As we had very little missing data, no specific missing-case analyses were conducted. We analyzed the data using SAS version 9.4 (SAS Institute). We conducted a hierarchical cluster analysis among the innovations using the SAS CLUSTER procedure.32 This allowed us to group innovations into the most homogeneous clusters possible, based on the number of criteria met. Thus, the hierarchical clustering analysis was aimed to create clusters of innovations but not to analyze existing clusters. The objective was to rank innovations in order of their scalability. We used the average linkage method, whereby the distance between 2 clusters is the average of the distances of all pairs of innovations, 1 in each cluster. 32,33 We estimated the optimal number of clusters using the pseudo F and pseudo t2 statistics.32

Ethics approval

Our study assessed research innovations and did not collect data for human participants; thus, ethics review was not needed.34

Results

Participants

Eleven of the 12 teams reported at least 1 evidence-based innovation (Figure 1). One team did not respond owing to lack of time and capacity. We received information for 33 innovations (median 3, range 1–8 per team). As shown in Table 1, 23 innovations focused on chronic diseases and/or access to resources (e.g., health care services, social services and complex care needs).

Figure 1:
Figure 1:

Flow chart showing identification of evidence-based innovations.

View this table:
Table 1:

Characteristics of the 33 evidence-based innovations reported by 11 teams

Types of evidence-based innovations

The majority (21) of the evidence-based innovations were health interventions (e.g., a community partnership program aiming to reduce the impact of vascular disease on the health of Canadians35), followed by analytical methods (4 innovations) (e.g., piloting an automated practice-based patient survey system by telephone or email), conceptual frameworks (4 innovations) (e.g., a guide to implementing integrated care), measures (3 innovations) (e.g., a 21-item measure for identifying people with multimorbidity in primary care) and strategies to build research capacity (1 innovation).

Scalability assessment dimensions

Use of a theory to develop the innovation (1 criterion) was the dimension in which scalability assessment criteria were most often met (n = 29) (e.g., the Consolidated Framework for Implementation Research), followed by impact (mean 22.3 [SD 5.6] innovations per dimension), setting (mean 21.7 [SD 8.5]), cost (mean 17.5 [SD 2.1]) and coverage (mean 14.0 [SD 4.1]) (Table 2). Within the coverage dimension, the criteria of reach, adoption and maintenance were frequently reported as not applicable (n = 12, 12 and 14, respectively) (Supplementary Table A1, Appendix 1), with the most commonly reported reason being that the innovation was “not an intervention.”

View this table:
Table 2:

Number of evidence-based innovations that met scalability assessment criteria

On average, 10 (SD 4) of the 16 scalability assessment criteria were met by the 33 innovations. Adoption was the least assessed criterion (9 innovations). The number of criteria that were met in the coverage dimension varied with the type of innovation (p = 0.005), with health interventions likely to meet the most criteria.

Ranking for scalability

Using hierarchical cluster analysis, we classified the 33 innovations into 3 groups (pseudo F = 73.5, pseudo t2 = 8.0): those whose scalability was ranked as high (n = 20), those whose scalability was ranked as medium (n = 11) and those whose scalability was ranked as low (n = 2) (Table 1). The mean number of scalability assessment criteria met for the 3 groups was 12 (SD 2), 7 (SD 2) and 1 (SD 1), respectively. A high ranking indicated that the team had collected diverse important information relevant to making a decision about the scale up of the innovation. Nine of the 11 responding teams ranked at least 1 of their innovations as having high potential for scale up (median 1, range 0−5 per team). Most (16) of the 20 high-ranking innovations were health interventions.

Interpretation

In this study, we explored the scalability of a large sample of evidence-based innovations in Canadian community-based primary health care. The majority of innovations were health interventions, and most ranked high for scalability. However, few of the innovations met the scalability assessment criteria relating to coverage, a dimension essential for scalability. These findings lead us to make the following observations.

First, the innovations that ranked highest for scalability were health interventions. For example, the pop-up health and community service event met all of the scalability assessment criteria.36 As the innovation had already been implemented and assessed in several different settings (e.g., a seniors’ centre, the Indigenous Early Intervention Organization, an elementary school), the team was likely to have collected a variety of relevant data. A crucial element of successful scale up is user-informed assessment of the environment to determine whether there is a fit between the target units and the innovation.16,37 A danger in health care system reform is the rolling-out of innovations as population-wide “one-size-fits-all” solutions. This project had clearly based its scaled-up implementation on individual and community-based needs.1,36

Second, innovations that were not health interventions were overall ranked lower in scalability. The diversity of innovations developed by the teams (from health interventions to methodological innovations) reflects the inherent complexity of community-based primary health care research, not all of which can be appropriately or easily scaled up. Teams whose innovations consisted of a conceptual framework or an analytical method, for example, were less concerned about widespread implementation of their innovations as they were not necessarily designed to directly improve health functioning or alleviate conditions.38 In addition, the diffusion of methodological innovation often takes a more complex, intuitive and organic path that is more horizontal than top-down. 8,39,40 However, there is value in assessing, adapting and scaling up methodological innovations for wider use by researchers,39,41–44 and there are too few opportunities for researchers to develop and experiment with methods for addressing social and clinical research questions.39,45 The usefulness of all scaled-up innovations will ultimately depend on the strength of the methodology developed.

Third, we found that coverage was the dimension in which scalability assessment criteria were least likely to be met, and adoption was the least assessed criterion. This may be in part because some innovations were not yet fully implemented or were designed to remain local rather than scalable. Lack of scalability frameworks to support research design and lack of sufficient time and resources could be other reasons. Moreover, some of the concepts relating to the science of scale up (e.g., adoption, adaptation, maintenance, spread and scale up itself) are as yet ill-defined and undertheorized in knowledge translation.46 Some of these concepts were unfamiliar to some of the stakeholders responding to our questionnaire. This highlights the need to refine and standardize terminology relating to scale-up science.2,47

Finally, it is difficult to compare our results with the literature, as we found few studies that analyzed primary health care innovations using scalability assessment criteria. The authors of a study assessing scaled-up public health interventions found that 25% of interventions had not been trialled, and, in 5%, efficacy had not been assessed, but they did not address coverage.48 The authors of a systematic review on scaling up evidence-based innovations in primary care noted that coverage was least likely to be assessed.2 Coverage of targeted users or settings is at the heart of scalability.2,13,17 For innovations to have a substantial impact, they need to be adopted by a large enough population over a sustained period.13 Developers of innovations can evaluate their programs using the Reach, Effectiveness, Adoption, Implementation, and Maintenance Qualitative Evaluation for Systematic Translation (RE-AIM QuEST) framework to specifically explore the coverage potential of their innovations.49,50

Limitations

First, in the interests of brevity, our questionnaire did not adequately explain every dimension of assessing scalability. Two teams contacted us for further explanation. Although the questionnaire was not validated, it was created through a group process (with 3 teams) and was pretested with at least 3 innovations. Second, we collected data on the existence of assessment criteria but not on the results of those assessments. Thus, based on our data, we cannot confidently label the innovations as scalable or not. However, our data clearly show that some teams collected important information that would facilitate scale up of their innovations. Moreover, feedback from the team members indicated that they found the experience constructive and informative for the purposes of future scale-up planning. Third, the questionnaires were completed by team stakeholders who, although best qualified to answer detailed questions about their projects, were also researchers in the study. This may have reduced objectivity. Also, all teams had end-user partners, and if these partners had completed questionnaires as well, they would have added an important perspective on the project’s scalability. Finally, scores obtained with the hierarchical cluster analysis are specific to the innovations used, which does not allow for generalization of results. As a first exercise to rank the scalability of evidence-based innovations in community-based primary health care in Canada, these findings will enable us to refine the language of scaling up, adapt the scalability assessment criteria, improve our questionnaire and validate it for future studies.

Conclusion

We explored scalability assessment criteria among a large sample of evidence-based innovations in Canadian community-based primary health care. Overall, teams had important information about their innovations that they could use to plan scale up. However, coverage is a critical dimension that remains largely unaddressed. Our findings provide concrete data on evaluating innovations for larger-scale use and contribute important new understandings of scalability assessment with relevance to a broad group of stakeholders. The questionnaire itself served as a knowledge-translation tool, motivating teams to talk with their research communities and decision-makers about scalability. Finally, our findings inform stakeholders about how and why scalability should be evaluated and reported as an outcome measure in project design and grant submission.

Acknowledgements

The authors thank the following team-nominated principal investigators: Marshall Godwin, Eva Grunfeld, Jeannie Haggerty, Stewart Harris, Janusz Kaczorowski, Moira Stewart and Walter Wodchis. They also thank the members of the teams for their contribution: ACCESS-MH (Atlantic Canada Children’s Effective Service Strategies in Mental Health), ACHRU (Aging, Community and Health Research Unit), CanIMPACT (Canadian Team to Improve Community-Based Cancer Care along the Continuum), C-ChAMP (Canadian Chronic Disease Awareness and Management Program), CircHSIT (Circumpolar Health System Innovation Team), FORGE AHEAD (Transformation of Indigenous Primary Healthcare Delivery), iCOACH (Implementing Integrated Care for Older Adults with Complex Health Needs), IMPACT (Innovative Models Promoting Access-to-Care Transformation), iPHIT (Innovation in Community-Based Primary Healthcare Supporting Transformation in the Health of First Nations and Rural/Remote Manitoba Communities), LHIV (Living with HIV Innovation Team), PACE in MM (Patient-Centred Innovations for Persons with Multimorbidity) and TRANSFORMATION (Improving the Science and Reporting of Performance in Primary Care). Finally, the authors thank Louisa Blair, English-language editor, for her kind help with the manuscript.

Footnotes

  • Competing interests: None declared.

  • This article has been peer reviewed.

  • Contributors: Ali Ben Charif, Kasra Hassani, Sabrina Wong, Hervé Zomahoun, Martin Fortin, Adriana Freitas, Kathryn Nicholson, Jenny Ploeg and France Légaré contributed to the study conception and design. Kasra Hassani and Sabrina Wong coordinated data collection. The 12 research teams contributed to data collection. Ali Ben Charif analyzed the data and drafted the manuscript, overseen by France Légaré. All of the authors contributed to the interpretation of data, revised the manuscript critically for important intellectual content, gave final approval of the version to be published and agreed to be accountable for all aspects of the work.

  • Funding: This study was funded by Canadian Institutes of Health Research grants awarded to the 12 research teams. Ali Ben Charif was funded by the Fonds de recherche du Québec − Santé. Ali Ben Charif and Hervé Zomahoun were funded by the Quebec Strategy for Patient-Oriented Research (SPOR) Support for People and Patient-Oriented Research and Trials (SUPPORT) unit. France Légaré holds a Tier 1 Canada Research Chair in Shared Decision Making and Knowledge Translation, and Adriana Freitas was funded by this chair.

  • Disclaimer: Only the authors have responsibility for the information provided or views expressed in this article.

  • Supplemental information: For reviewer comments and the original submission of this manuscript, please see www.cmajopen.ca/content/6/4/E520/suppl/DC1.

References

  1. ↵
    1. Fooks C
    (2003) Implementing primary care reform in Canada: barriers and facilitators (McGill-Queen’s University Press, Montréal) Available: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.502.6522&rep=rep1&type=pdf. accessed 2018 Apr. 26.
  2. ↵
    1. Ben Charif A,
    2. Zomahoun HTV,
    3. LeBlanc A,
    4. et al.
    (2017) Effective strategies for scaling up evidence-based practices in primary care: a systematic review. Implement Sci 12:139.
    OpenUrlCrossRefPubMed
    1. Fixsen D,
    2. Blase K,
    3. Metz A,
    4. et al.
    (2013) Statewide implementation of evidence-based programs. Except Child 79:213–30.
    OpenUrl
  4. ↵
    1. Milat AJ,
    2. Bauman A,
    3. Redman S
    (2015) Narrative review of models and success factors for scaling up public health interventions. Implement Sci 10:113.
    OpenUrlPubMed
  5. ↵
    1. Haynes B,
    2. Haines A
    (1998) Barriers and bridges to evidence based clinical practice. BMJ 317:273–6.
    OpenUrlFREE Full Text
    1. Kristensen N,
    2. Nymann C,
    3. Konradsen H
    (2016) Implementing research results in clinical practice — the experiences of healthcare professionals. BMC Health Serv Res 16:48.
    OpenUrlPubMed
  7. ↵
    1. Bégin M,
    2. Eggertson L,
    3. Macdonald N
    (2009) A country of perpetual pilot projects. CMAJ 180:1185, E88–9.
    OpenUrlFREE Full Text
  8. ↵
    1. Shaw J,
    2. Tepper J,
    3. Martin D
    (2018) From pilot project to system solution: innovation, spread and scale for health system leaders. BMJ Leader 2:87–90.
    OpenUrlAbstract/FREE Full Text
    1. Massoud MR,
    2. Donohue KL,
    3. McCannon CJ
    (2010) Options for large-scale spread of simple high-impact interventions [technical report] (USAID Health Care Improvement Project), (University Research Co, LLC), (United States Agency for International Development), (World Health Organization), (Harvard School of Public Health, Chevy Chase (MD)), (Chevy Chase (MD)), (Washington (DC)), (Geneva), (Boston) Available: https://www.urc-chs.com/resources/options-large-scale-spread-simple-high-impact-interventions. accessed 2018 Apr. 26.
    1. Eaton J,
    2. McCay L,
    3. Semrau M,
    4. et al.
    (2011) Scale up of services for mental health in low-income and middle-income countries. Lancet 378:1592–603.
    OpenUrlCrossRefPubMed
  11. (2005) How-to guide: sustainability and spread (Institute for Healthcare Improvement, Boston) Available: www.ihi.org/resources/pages/tools/howtoguidesustainabilityspread.aspx. accessed 2018 Apr. 26.
    1. Albury D,
    2. Beresford T,
    3. Dew S,
    4. et al.
    (2018) Against the odds: successfully scaling innovation in the NHS (Innovation Unit and The Health Foundation, London (UK)) Available: http://kingsfund.blogs.com/health_management/2018/01/against-the-odds-successfully-scaling-innovation-in-the-nhs.html. accessed 2018 Feb 8.
  13. ↵
    1. Milat AJ,
    2. Newson R,
    3. King L,
    4. et al.
    (2016) A guide to scaling up population health interventions. Public Health Res Pract 26:e2611604.
    OpenUrl
  14. ↵
    (2010) Nine steps for developing a scaling-up strategy (World Health Organization, Geneva) Available: www.who.int/reproductivehealth/publications/strategic_approach/9789241500319/en/. accessed 2018 Apr. 26.
    1. Barker PM,
    2. Reid A,
    3. Schall MW
    (2016) A framework for scaling up health interventions: lessons from large-scale improvement initiatives in Africa. Implement Sci 11:12.
    OpenUrlCrossRefPubMed
  16. ↵
    1. Bradley EH,
    2. Curry LA,
    3. Taylor LA,
    4. et al.
    (2012) A model for scale up of family health innovations in low-income and middle-income settings: a mixed methods study. BMJ Open 2:e000987.
    OpenUrlAbstract/FREE Full Text
  17. ↵
    1. Fixsen DL,
    2. Blase KA,
    3. Fixsen AA
    (2017) Scaling effective innovations. Criminol Public Policy 16:487–99.
    OpenUrl
  18. ↵
    1. Ploeg J,
    2. Markle-Reid M,
    3. Davies B,
    4. et al.
    (2014) Spreading and sustaining best practices for home care of older adults: a grounded theory study. Implement Sci 9:162.
    OpenUrlCrossRefPubMed
    1. Rocker GM,
    2. Amar C,
    3. Laframboise WL,
    4. et al.
    (2017) Spreading improvements for advanced COPD care through a Canadian collaborative. Int J Chron Obstruct Pulmon Dis 12:2157–64.
    OpenUrl
  20. ↵
    1. Gaziano TA,
    2. Galea G,
    3. Reddy KS
    (2007) Scaling up interventions for chronic disease prevention: the evidence. Lancet 370:1939–46.
    OpenUrlCrossRefPubMed
  21. ↵
    Community-based primary health care: research profiles (Canadian Institutes of Health Research, Ottawa) (modified 2018 May 8). Available: www.cihrirsc.gc.ca/e/50370.html#it12. accessed 2018 Apr. 26.
  22. ↵
    1. Wong ST,
    2. Langton JM,
    3. Katz A,
    4. et al.
    Promoting cross-jurisdictional primary health care research: implementing 12 community based primary health care teams across Canada. Prim Health Care Res Dev.
  23. ↵
    1. Rogers EM
    (1995) Diffusion of innovations (Simon and Schuster, New York) Available: https://books.google.ca/books?id=v1ii4QsB7jIC&printsec=frontcover#v=onepage&q&f=false. accessed 2018 Oct. 2. 4th ed.
  24. ↵
    1. Flay BR,
    2. Biglan A,
    3. Boruch RF,
    4. et al.
    (2005) Standards of evidence: criteria for efficacy, effectiveness and dissemination. Prev Sci 6:151–75.
    OpenUrlCrossRefPubMed
  25. ↵
    International Classification of Health Interventions (ICHI) (World Health Organization, Geneva) (updated 2016 Aug. 18). Available: www.who.int/classifications/ichi/en/. accessed 2018 Apr. 26.
  26. ↵
    1. Deblaise D,
    2. Hernot X,
    3. Maurine P
    Proceedings 2006 IEEE International Conference on Robotics and Automation (2006 May 15–19, Orlando (FL)), A systematic analytical method for PKM stiffness matrix calculation, pp 4213–9.
  27. ↵
    1. Charmaz K,
    2. Belgrave LL
    (2015) The Blackwell encyclopedia of sociology, Grounded theory (John Wiley & Sons, Hoboken (NJ)) Available: http://onlinelibrary.wiley.com/doi/10.1002/9781405165518.wbeosg070.pub2/abstract. accessed 2018 Feb. 14.
  28. ↵
    1. Eisenhart M
    Proceedings of the Thirteenth Annual Meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education (1991 Oct. 16–19, Blacksburg (VA)), Conceptual frameworks for research circa 1991: ideas from a cultural anthropologist; implications for mathematics education researchers, ed Underhill RG, Available: http://nepc.colorado.edu/files/Eisenhart_ConceptualFrameworksforResearch.pdf, pp 202–19. accessed 2018 Jan. 18.
  29. ↵
    (Jan 28, 2010) Health indicators overview: questions and answers (US National Library of Medicine, Bethesda (MD)) (updated 2018 July 3). Available: https://www.nlm.nih.gov/nichsr/healthindicators/Health_Indicators_Overview_Q_and_A.html. accessed 2018 Jan. 18.
  30. ↵
    (2017) Understanding capacity-building/capacity development: a core concept of development policy (European Parliament, Brussels) Available: www.europarl.europa.eu/thinktank/en/document.html?reference=EPRS_BRI(2017)599411. accessed 2018 Jan. 18.
  31. ↵
    1. Cooke J
    (2005) A framework to evaluate research capacity building in health care. BMC Fam Pract 6:44.
    OpenUrlCrossRefPubMed
  32. ↵
    (2016) SAS/STAT® 142 user’s guide: the CLUSTER procedure (SAS Institute, Cary (NC)) Available: https://support.sas.com/documentation/onlinedoc/stat/142/cluster.pdf. accessed 2018 Apr. 26.
  33. ↵
    1. Sokal R,
    2. Michener C
    (1958) A statistical method for evaluating systematic relationship. Univ Kans Sci Bull 38:1409–38.
    OpenUrl
  34. ↵
    1. Fletcher J
    (2015) Ethical approval for all studies involving human participants. CMAJ 187:91.
    OpenUrlFREE Full Text
  35. ↵
    1. Agarwal G,
    2. Angeles R,
    3. Pirrie M,
    4. et al.
    (2017) Effectiveness of a community paramedic-led health assessment and education initiative in a seniors’ residence building: the Community Health Assessment Program through Emergency Medical Services (CHAP-EMS). BMC Emerg Med 17:8.
    OpenUrlPubMed
  36. ↵
    1. Scott C,
    2. Barnes J,
    3. Mallard R,
    4. et al.
    Proceedings of the Forty-fifth North American Primary Care Research Group meeting (2017 Nov. 17–21, Montréal), Evaluating a pop-up health and community event in Lethbridge, Alberta, Canada (North American Primary Care Research Group, Leawood (KS)) Available: www.napcrg.org/Conferences/AnnualMeeting/EducationEvents/SearchEducationalSessions?m=6&s=21149. accessed 2018 Mar. 29.
  37. ↵
    1. Bradley EH
    (2018) Community health workers are effective but how to scale-up? Am J Public Health 108:1129–30.
    OpenUrl
  38. ↵
    1. Rycroft-Malone J,
    2. Seers K,
    3. Titchen A,
    4. et al.
    (2004) What counts as evidence in evidence-based practice? J Adv Nurs 47:81–90.
    OpenUrlCrossRefPubMed
  39. ↵
    1. Wiles R,
    2. Crow G,
    3. Pain H
    (2011) Innovation in qualitative research methods: a narrative review. Qual Res 11:587–604.
    OpenUrlCrossRef
  40. ↵
    1. Rogers EM
    (2003) Diffusion of innovations (Free Press, New York), 5th ed.
  41. ↵
    1. Rapport F,
    2. Clay‐Williams R,
    3. Churruca K,
    4. et al.
    (2018) The struggle of translating science into action: foundational concepts of implementation science. J Eval Clin Pract 24:117–26.
    OpenUrlPubMed
    1. Wainwright DW,
    2. Waring TS
    (2007) The application and adaptation of a diffusion of innovation framework for information systems research in NHS general medical practice. J Inf Technol 22:44–58.
    OpenUrl
    1. Bobrow K,
    2. Farmer A,
    3. Cishe N,
    4. et al.
    (2018) Using the Medical Research Council framework for development and evaluation of complex interventions in a low resource setting to develop a theory-based treatment support intervention delivered via SMS text message to improve blood pressure control. BMC Health Serv Res 18:33.
    OpenUrl
  44. ↵
    1. Harting J,
    2. Rutten GM,
    3. Rutten ST,
    4. et al.
    (2009) A qualitative application of the diffusion of innovations theory to examine determinants of guideline adherence among physical therapists. Phys Ther 89:221–32.
    OpenUrlAbstract/FREE Full Text
  45. ↵
    1. Taylor C,
    2. Coffey A
    (2008) Innovation in qualitative research methods: possibilities and challenges, School of Social Sciences Working Papers series vol 121 (Cardiff University, Cardiff UK).
  46. ↵
    1. Ilott I,
    2. Gerrish K,
    3. Pownall S,
    4. et al.
    (2013) Exploring scale-up, spread, and sustainability: an instrumental case study tracing an innovation to enhance dysphagia care. Implement Sci 8:128.
    OpenUrlCrossRefPubMed
  47. ↵
    1. Fixsen AAM
    (2009) Defining scaling up across disciplines: an annotated bibliography (Portland State University, Portland (OR)) Available: https://pdfs.semanticscholar.org/57fc/57337970444b7bfc536adee7d0791a36471b.pdf. accessed 2018 Sept. 28.
  48. ↵
    1. Indig D,
    2. Lee K,
    3. Grunseit A,
    4. et al.
    (2017) Pathways for scaling up public health interventions. BMC Public Health 18:68.
    OpenUrlCrossRefPubMed
  49. ↵
    1. Forman J,
    2. Heisler M,
    3. Damschroder LJ,
    4. et al.
    (2017) Development and application of the RE-AIM QuEST mixed methods framework for program evaluation. Prev Med Rep 6:322–8.
    OpenUrlPubMed
  50. ↵
    1. Glasgow RE,
    2. Vogt TM,
    3. Boles SM
    (1999) Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 89:1322–7.
    OpenUrlCrossRefPubMed
Back to top

In this issue

CMAJ Open: 6 (4)
Vol. 6, Issue 4
1 Oct 2018

Article tools

Respond to this article
Print
Download PDF
Article Alerts
Email Article
Citation Tools
Assessment of scalability of evidence-based innovations in community-based primary health care: a cross-sectional study
Ali Ben Charif, Kasra Hassani, Sabrina T. Wong, Hervé Tchala Vignon Zomahoun, Martin Fortin, Adriana Freitas, Alan Katz, Claire E. Kendall, Clare Liddy, Kathryn Nicholson, Bojana Petrovic, Jenny Ploeg, France Légaré
Oct 2018, 6 (4) E520-E527; DOI: 10.9778/cmajo.20180143
Digg logo Reddit logo Twitter logo Facebook logo Google logo Mendeley logo

Collections