The challenges of measuring quality-of-care indicators in rural emergency departments: a cross-sectional descriptive study
==========================================================================================================================

* Géraldine Layani
* Richard Fleet
* Renée Dallaire
* Fatoumata K. Tounkara
* Julien Poitras
* Patrick Archambault
* Jean-Marc Chauny
* Mathieu Ouimet
* Josée Gauthier
* Gilles Dupuis
* Alain Tanguay
* Jean-Frédéric Lévesque
* Geneviève Simard-Racine
* Jeannie Haggerty
* France Légaré

## Abstract

**Background:** Evidence-based indicators of quality of care have been developed to improve care and performance in Canadian emergency departments. The feasibility of measuring these indicators has been assessed mainly in urban and academic emergency departments. We sought to assess the feasibility of measuring quality-of-care indicators in rural emergency departments in Quebec.

**Methods:** We previously identified rural emergency departments in Quebec that offered medical coverage with hospital beds 24 hours a day, 7 days a week and were located in rural areas or small towns as defined by Statistics Canada. A standardized protocol was sent to each emergency department to collect data on 27 validated quality-of-care indicators in 8 categories: duration of stay, patient safety, pain management, pediatrics, cardiology, respiratory care, stroke and sepsis/infection. Data were collected by local professional medical archivists between June and December 2013.

**Results:** Fifteen (58%) of the 26 emergency departments invited to participate completed data collection. The ability to measure the 27 quality-of-care indicators with the use of databases varied across departments. Centres 2, 5, 6 and 13 used databases for at least 21 of the indicators (78%-92%), whereas centres 3, 8, 9, 11, 12 and 15 used databases for 5 (18%) or fewer of the indicators. On average, the centres were able to measure only 41% of the indicators using heterogeneous databases and manual extraction. The 15 centres collected data from 15 different databases or combinations of databases. The average data collection time for each quality-of-care indicator varied from 5 to 88.5 minutes. The median data collection time was 15 minutes or less for most indicators.

**Interpretation:** Quality-of-care indicators were not easily captured with the use of existing databases in rural emergency departments in Quebec. Further work is warranted to improve standardized measurement of these indicators in rural emergency departments in the province and to generalize the information gathered in this study to other health care environments.

Providing equitable quality emergency care to rural citizens in a vast country with limited financial and human resources is a great challenge. Twenty percent of Quebec's population lives in rural regions,1 and rural emergency departments in the province receive an average of 19 000 visits per year.2-6 Given the limited access to diagnostic services, family doctors and other specialists in rural areas, rural emergency departments constitute an essential safety net for the rural population.2-4,7 Furthermore, in an effort to limit the inherent costs related to emergency departments in rural regions, several Canadian provinces have reduced or regionalized these services.8-10 As a result, numerous hospitals have been forced to reduce services or to close altogether.11 The impact of this situation on the quality of care is not well known. Timely attempts to measure and monitor quality of care in rural emergency departments are thus warranted.

The publication *Development of a Consensus on Evidence-Based Quality of Care Indicators for Canadian Emergency Departments*12 takes us a step closer to this objective. This document, which was published in 2010, was created by a panel of 24 Canadian experts including managers, clinicians, emergency medicine researchers, health information specialists and government representatives. The selected indicators are related to interventions for life-threatening conditions often treated in emergency departments, including myocardial infarction, stroke, sepsis, asthma and several pediatric problems related to infection (Appendix 1, available at [www.cmajopen.ca/content/4/3/E398/suppl/DC1](http://www.cmajopen.ca/content/4/3/E398/suppl/DC1)). Theseindicators were developed through an extensive modified Delphi process and are now considered the reference standard in Canada for evaluating quality of care in emergency departments.

It is expected that quality-of-care indicators will allow clinical staff, administrators and researchers to identify areas where improvement in clinical care is most needed, establish benchmarks and compare care across emergency departments in a valid and reliable way.13 These indicators could have a substantial impact on the quality of care provided to rural citizens.12,13 The implementation and regular follow-up of indicators could help standardize access to quality care in rural areas, identify the needs of the population and improve organization of care. The end goal is for rural patients to receive the standard treatment for their medical condition, rather than care that simply reflects the resources available in the area.

However, there are practical limitations related to measuring quality-of-care indicators in rural emergency departments. First, information on indicators may not be available in clinical databases in every rural emergency department. Second, collection of data on certain priority indicators could be difficult in rural establishments owing to lack of resources.14

The primary objective of this study was to investigate the feasibility of measuring the quality-of-care indicators defined by Schull and colleagues12 in rural emergency departments in Quebec and to identify potential barriers to implementing the indicators. The study is a substudy of a larger cross-sectional multicentre research project4,15 (Figure 1). 

![Figure 1](http://www.cmajopen.ca/https://www.cmajopen.ca/content/cmajo/4/3/E398/F1.medium.gif)

[Figure 1](http://www.cmajopen.ca/content/4/3/E398/F1)

Figure 1 
Flow chart of centres participating in project on rural emergency departments (EDs) in Quebec and use of provincial emergency department management guide.4,15,16

## Setting And Study Design

Rural emergency departments in Quebec were previously identified.17,18 Emergency departments were selected if they had medical coverage with hospital beds 24 hours a day, 7 days a week and were situated in a rural region as per the Statistics Canada definition.17 Using these criteria, we identified 26 rural emergency departments in the province.

## Source Of Data

Data were collected via email from June to December 2013. We described and explained the 27 indicators in a Microsoft Word document (Appendix 2, available at [www.cmajopen.ca/content/4/3/E398/suppl/DC1](http://www.cmajopen.ca/content/4/3/E398/suppl/DC1)) and created an Excel spreadsheet to standardize data collection from the emergency departments' patient databases. To ensure standardized measurement of the indicators, our institution's archivists pretested the protocol. Conference calls were held with the head of medical archives in each of the participating emergency departments to introduce the study and identify the person in charge of data collection at each emergency department. The data collection protocols were emailed to the medical archives specialists at each of the invited centres. Data were collected from databases and patient medical files by the person in charge at each participating emergency department. A graduate student/physician (G.L.) and a research nurse followed up weekly via telephone or email to ensure that proper procedures were followed (as described in the Word document).

### Statistical analysis

The emergency departments were coded with a number from 1 to 15 as per protocol. We analyzed the data using descriptive statistics (mean, median, proportion). All analyses were conducted with SAS software.

The study was approved by the research ethics committee at the research centre of the Hôtel-Dieu de Lévis, a university-affiliated hospital in Lévis, Quebec.

## Results

### Sample size and participation rate

Seven of the 26 previously identified emergency departments declined to participate in this phase of the project, mainly owing to lack of human resources. Of the remaining 19 emergency departments, 4 were excluded for failure to complete data collection owing to lack of time and personnel. The final sample included 15 emergency departments, for a participation rate of 58% (Figure 1).

### Measurement of quality-of-care indicators

All the indicators in the duration of stay, patient safety, cardiology and sepsis/infection categories were measurable in 7 or more (47%-73%) of the emergency departments (Table 1). For the remaining categories, the indicators were measurable in 6 (40%) or fewer emergency departments. One indicator in the respiratory care category, QCI 11.1 (number of patients with asthma [by age group] who received corticosteroids at the emergency department and at release [if released]), was not measurable in any of the emergency departments.

View this table:
[Table 1:](http://www.cmajopen.ca/content/4/3/E398/T1)

Table 1:   Measurability of individual quality-of-care indicators12

### Database use

The ability to measure the 27 quality-of-care indicators with the use of databases varied across emergency departments. Centres 2, 5, 6 and 13 used databases for at least 21 of the indicators (78%-92%), whereas centres 3, 8, 9, 11, 12 and 15 used databases for 5 (18%) or fewer of the indicators (Table 2). The average number of indicators that could be measured was 11.2 (41%). The centres searched 15 different databases and 15 combinations of databases (in cases in which the archivist was willing to use 2 or more databases for 1 or more indicators) (Appendix 3, available at [www.cmajopen.ca/content/4/3/E398/suppl/DC1](http://www.cmajopen.ca/content/4/3/E398/suppl/DC1)).

View this table:
[Table 2:](http://www.cmajopen.ca/content/4/3/E398/T2)

Table 2:   Frequency of database use to measure quality-of-care indicators

### Time required for measurement

The average total time required to measure the 27 indicators ranged from 5 to 88.5 minutes (data not shown). The median data collection time for each indicator was 15 minutes or less in most cases. However, for indicators 6, 9, 11.5 and 11.6, the median time was 30 minutes or more (Table 3).

View this table:
[Table 3:](http://www.cmajopen.ca/content/4/3/E398/T3)

Table 3:   Median time to measure quality-of-care indicators

## Interpretation

We found that the existing emergency department databases did not permit measurement of several established evidence-based quality-of-care indicators. Specifically, the centres were able to measure only 41% of the indicators using heterogeneous databases and manual extraction. The 15 participating centres collected data from 15 different databases or combinations of databases, and the process of extracting the data was time-consuming. Our results suggest that the quality of databases and access to them are the most important feasibility considerations. Information on several different indicators was inaccessible, and archivists were obliged to conduct manual searches of paper patient files to extract data on indicators. Manual consultation of paper files is resource-intensive and in our study required the use of several intermediaries, which discouraged some participants, resulting in failure to measure several indicators. Furthermore, even for centres with access to databases, the information the databases contained was not useful for measuring the quality-of-care indicators, and the archivists had to resort to considerable data manipulation to measure the indicators.

Our results are consistent with those of Schull and colleagues,14 whose feasibility assessment in urban centres showed that 13 (27%) of 48 quality-of-care indicators could be measured with the use of current data elements in the Canadian Institute for Health Information's National Ambulatory Care Reporting System (NACRS) or the NACRS plus linkage with other administrative databases such as the Discharge Abstracts Database or death records. The 13 indicators did, however, include some higher-priority indicator categories for emergency department operations, such as patient safety and sepsis/infection.12 In an earlier study, Lindsay and colleagues13 found limited feasibility of calculating quality-of-care indicators using a routinely collected data set: of 29 indicators, only 8 were captured owing to lack of sufficient specificity within the NACRS and International Classification of Diseases, 9th revision coding systems to satisfy the operational definitions and owing to the need to link the emergency department visit to inpatient databases. While the Ontario health care system is recognized for the quality of its databases, such as those of the Institute for Clinical Evaluative Sciences, Toronto, other health care systems face challenges in measuring well-established quality-of-care indicators, especially those in small rural settings with limited resources/databases.

### Limitations

Because of limited resources, we did not plan or conduct interrater reliability assessments of the capture of quality-of-care indicators conducted by the emergency department medical archivists. We initially thought this work would be straightforward with existing databases and provided a protocol to capture the indicators. This assumption was incorrect and is hence a major limitation of this study. Furthermore, because of the difficulties in collecting data experienced by archivists, the data collected was incomplete, which limited the use of multivariate statistical analyses. We could not calculate any correlations to determine whether a relation existed between the databases used and the quality of the data collected.

### Conclusion

As presently defined, quality-of-care indicators were not easily captured with the use of existing databases in rural emergency departments in Quebec. Indicators concerning pediatrics, respiratory care and stroke were the most difficult to measure. Further work is warranted to improve standardized measurement of quality indicators in rural emergency departments in the province and to generalize the information gathered in this study to other health care environments.

### Supplemental information

For reviewer comments and the original submission of this manuscript, please see [www.cmajopen.ca/content/4/3/E398/suppl/DC1](http://www.cmajopen.ca/content/4/3/E398/suppl/DC1)

## Acknowledgements

The authors thank the rural emergency department staff across Quebec who participated in the study. The authors also thank Louisa Blair for editing assistance and Luc Lapointe for help formatting the manuscript.

## Footnotes

*   **Competing interests:** None declared.This article has been peer reviewed.

*   **Contributors:** Géraldine Layani drafted and revised the manuscript with Richard Fleet as part of her master's thesis under the supervision of Richard Fleet. Richard Fleet conceived and designed the study. Renée Dallaire, Julien Poitras, Patrick Archambault, Jean-Marc Chauny, Mathieu Ouimet, Josée Gauthier, Gilles Dupuis, Alain Tanguay, Jean-Frédéric Lévesque, Geneviève Simard-Racine, Jeannie Haggerty and France Légaré contributed to study design. Fatoumata Tounkara contributed significantly to drafting and revising the manuscript. All of the authors gave approval of the final version to be published and agreed to act as guarantors of the work.

*   **Funding:** This research project was supported by the Fonds de Recherche du Québec - Santé (grant number 22481) and the Public Health and the Agricultural Rural Ecosystem scholarship program. Géraldine Layani received a master's award from the University of Saskatchewan.

*   **Disclaimer:** To promote transparency, *CMAJ Open* publishes the names of public institutions in research papers that compare their performance or quality of care with that of other institutions. The authors of this paper have not linked individual institutions to outcome data. We decided to publish this paper without this information, as we felt that it was more important to communicate the finding that variation exists between institutions in the ability to measure quality-of-care indicators than to insist on transparency. -Editor, *CMAJ Open*

## References

1.  Ottawa: Statistics Canada (2011) Population, urban and rural, by province and territory (Quebec). Available[www.statcan.gc.ca/tables-tableaux/sum-som/l01/cst01/demo62f-eng.htm](http://www.statcan.gc.ca/tables-tableaux/sum-som/l01/cst01/demo62f-eng.htm). accessed 2014 Nov. 18.
    
    

2.  Fleet R, Poitras J, Maltais-Gigu, et al., re J, et al. (2013) A descriptive study of access to services in a random sample of Canadian rural emergency departments. BMJ Open 3:e003876.
    
    [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYm1qb3BlbiI7czo1OiJyZXNpZCI7czoxMjoiMy8xMS9lMDAzODc2IjtzOjQ6ImF0b20iO3M6MjA6Ii9jbWFqby80LzMvRTM5OC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

3.  Fleet R, Pelletier C, Marcoux J, et al. (2015) Differences in access to services in rural emergency departments of Quebec and Ontario. PLoS One 10:e0123746.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1371/journal.pone.0123746&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=25874948&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE398.atom) 

4.  Fleet R, Poitras J, Archambault P, et al. (2015) Portrait of rural emergency departments in Québec and utilization of the provincial emergency department management guide: cross sectional survey. BMC Health Serv Res 15:572.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1186/s12913-015-1242-0&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=26700302&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE398.atom) 

5.  Fleet R, Archambault P, Plant J, et al. (2013) Access to emergency care in rural Canada: Should we be concerned? CJEM 15:191–3.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=23777988&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE398.atom) 

6.  Adams GL, Campbell PT, Adams JM, et al. (2006) Effectiveness of prehospital wireless transmission of electrocardiograms to a cardiologist via hand-held device for patients with acute myocardial infarction (from the Timely Intervention in Myocardial Emergency, NorthEast Experience [TIME-NE]). Am J Cardiol 98:1160–4.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/j.amjcard.2006.05.042&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=17056318&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE398.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000241982200007&link_type=ISI) 

7.  Fatovich DM, Phillips M, Langford SA, et al. (2011) A comparison of metropolitan vs rural major trauma in Western Australia. Resuscitation 82:886–90.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/j.resuscitation.2011.02.040&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=21481512&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE398.atom) 

8.  Fleet R, Plant J, Ness R, et al. (2013) Patient advocacy by rural emergency physicians after major service cuts: the case of Nelson, BC. Can J Rural Med 18:56–61.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=23566863&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE398.atom) 

9.  Thompson JM, McNair NL (1995) Health care reform and emergency outpatient use of rural hospitals in Alberta, Canada. J Emerg Med 13:415–21.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/0736-4679(95)00014-2&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=7673640&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE398.atom) 

10. Victoria: Ministries of Health Services and Health Planning (2002) Standards of accessibility and guidelines for provision of sustainable acute care services by health authorities.
    
    

11. Collier R (2010) Is regionalization working? CMAJ 182:331–2.
    
    [FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czo5OiIxODIvNC8zMzEiO3M6NDoiYXRvbSI7czoyMDoiL2NtYWpvLzQvMy9FMzk4LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

12. Schull MJ, Hatcher CM, Guttmann A, et al., Toronto: Institute for Clinical Evaluative Sciences (2010) Development of a consensus on evidence-based quality of care indicators for Canadian emergency departments: ICES investigative report.
    
    

13. Lindsay P, Schull M, Bronskill S, et al. (2002) The development of indicators to measure the quality of clinical care in emergency departments following a modified-Delphi approach. Acad Emerg Med 9:1131–9.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1197/aemj.9.11.1131&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=12414461&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE398.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000179203100012&link_type=ISI) 

14. Schull MJ, Guttmann A, Leaver CA, et al. (2011) Prioritizing performance measurement for emergency department care: consensus on evidence-based quality of care indicators. CJEM 13:300–9, E28–43.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.2310/8000.2011.110334&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=21955411&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE398.atom) 

15. Fleet R, Archambault P, Légaré F, et al. (2013) Portrait of rural emergency departments in Quebec and utilisation of the Quebec Emergency Department Management Guide: a study protocol. BMJ Open 3:e002961.
    
    [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYm1qb3BlbiI7czo1OiJyZXNpZCI7czoxMToiMy80L2UwMDI5NjEiO3M6NDoiYXRvbSI7czoyMDoiL2NtYWpvLzQvMy9FMzk4LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

16. Ministère de la Santé et des Services sociaux du Québec, Québec: Gouvernement du Québec (2006) Guide de gestion de l'urgence.
    
    

17. du Plessis V, Beshiri R, Bollman RD, et al. (2001) Definitions of rural. Rural Small Town Can Anal Bull 3:1–17.
    
    

18. Ottawa: Association canadienne des soins de santé (2009) Guide des établissements de soins de santé du Canada 2009/2010.
    
    

*   Copyright 2016, Joule Inc. or its licensors