Frequency of and variation in low-value care in primary care: a retrospective cohort study
==========================================================================================

* Ciara Pendrith
* Meghan Bhatia
* Noah M. Ivers
* Graham Mecredy
* Karen Tu
* Gillian A. Hawker
* Susan B. Jaglal
* Lynn Wilson
* Kimberly Wintemute
* Richard H. Glazier
* Wendy Levinson
* R. Sacha Bhatia

## Abstract

**Background:** Low-value care, defined as care with a lack of benefit, can lead to higher health care costs, inconvenience to patients and, in some cases, harm to patients. The objectives of this study are to conduct exploratory analyses to understand how frequently selected low-value tests are ordered, to assess the degree of variation in ordering that exists across regions and practices, and to identify services that may warrant further investigation and targeted interventions.

**Methods:** We conducted a population-based retrospective cohort study using administrative health care databases from Ontario to identify rates of use of the following low-value services between fiscal years 2008/09 and 2012/13: computed tomography (CT) or magnetic resonance imaging (MRI) after a diagnosis of low back pain, Papanicolaou testing in women less than 21 years of age or older than 69 years of age and repeated dual-energy X-ray absorptiometry (DEXA) scanning within 2 years of an index scan. Regional and practice-level rates were calculated. Bivariate analyses were conducted to explore associations between patient factors and repeat DEXA scans.

**Results:** Repeated DEXA scans were the most common service (21.0%), whereas cervical cancer screening among women less than 21 years of age or older than 69 years of age (8.0%) and CT or MRI imaging for low back pain (4.5%) were less common. There was substantial variation across practices with rates of repeated DEXA scans, ranging from 4.0% to 54.9%, and cervical cancer screening, ranging from 0.9% to 35.2%. Patients with a high-risk index DEXA were more likely to receive a repeat scan (28.1%) than those with a baseline (8.9%) or low-risk (8.1%) scan.

**Interpretation:** There is significant, practice-level variation in the frequency of low-value testing for DEXA scans, back imaging and cervical cancer screening. There is a particular need for interventions that aim to reduce unnecessary DEXA scans.

Low-value care, defined as care in which there is a lack of benefit, or in which the benefits are outweighed by the potential risks, can lead to higher health care costs, patient inconvenience and, in some cases, harm to patients.1,2 There is growing recognition that low-value care is common in health systems around the world.3 The Institute of Medicine estimates that up to 30% of medical care may be classified as low-value care.1

The Choosing Wisely campaign is a grass roots effort to address the issue of low-value care that launched in the United States in 2012.4 The campaign, which has been adopted by 18 countries, aims to change practice by harnessing physician leadership, increasing awareness regarding low-value tests, procedures and treatments, and by emphasizing the inherent risks to patients.3 The campaign launched in Canada in 2014, with 8 specialty societies releasing "top 5" lists of common clinical practices that are considered low-value care.5 One of the original specialties to participate was the College of Family Physicians of Canada, which now endorses 11 evidence-based Choosing Wisely recommendations.6

The objectives of this study are to conduct exploratory analyses to understand how frequently selected low-value tests are ordered, to assess the degree of variation in ordering that exists across regions and practices.7

## Methods

### Setting

This study was conducted in Ontario between fiscal years 2008 and 2013. Ontario residents with coverage under the Ontario Health Insurance Plan (OHIP) were potentially included in the study. The study period was chosen to get an estimate of baseline rates of use of low-value care before the publication of the Choosing Wisely Canada recommendations in 2014 and 2015.

### Study design and data sources

We conducted a retrospective cohort study in Ontario, Canada, using linked population-based administrative health care databases held at the Institute for Clinical Evaluative Sciences (ICES). The data sets were linked using unique encoded identifiers and analyzed at ICES. The Registered Persons Database contains demographic information on all Ontario residents eligible for the Ontario Health Insurance Plan (OHIP). The OHIP claims database contains all billing claims made by Ontario physicians. Client Agency Program Enrolment tables were used to identify patients rostered to primary care physicians.

We selected 3 Choosing wisely recommendations relevant to primary care for investigation: imaging for low back pain in the absence of red flags,6,8,9 cervical cancer screening for women under the age of 21 years or more than 69 years of age6,8,10 and repeat dual X-ray absorptiometry (DEXA) scans more often than every 2 years.11,12 These recommendations were chosen because we believed them to be commonly ordered in primary care and accurately measurable from administrative data through the use of previously published search algorithms.13-15

### Cohort selection

The study cohort was selected from Ontario patients with a valid provincial OHIP number who met eligibility criteria for 1 or more study recommendation between Apr. 1, 2008, and Mar. 31, 2013. Drawing from indicator definitions in the literature,13-17 we developed algorithms using physician claims and hospital encounter data to identify the cohorts for each study recommendation. In brief, we selected patients meeting eligibility criteria for 1 or more cohort: adults with a diagnosis of low back pain without red flags from a family physician based on methods developed by ICES;13 women aged 13-21 years or 69-105 years with no prior diagnoses of gynecological cancer or hysterectomy based on a validated algorithm to identify women who underwent screening for cervical cancer in Ontario;15 or adults 40 years of age and older who received a DEXA scan based on billing claims previously described by Jaglal and colleagues and previously validated.14,18This cohort represents women for whom screening for cervical cancer is not recommended according to both Choosing Wisely Canada recommendations and Cancer Care Ontario.20

### Covariates

We collected patient age, sex, rurality and neighbourhood income quintile. For patients in the repeat DEXA scan cohort, we collected information on the type of index scan from the fee schedule. A patient's first DEXA is a "baseline scan." Thereafter, patients are classified by risk - patients with osteoporosis, osteopenia or considered at high risk for accelerated bone loss on a previous DEXA receive "high-risk scans" and the remaining patients receive "low-risk scans." There are no differences in the amount payable across risk levels. We determined whether patients had a regular family physician using enrolment status in the Client Agency Program Enrolment tables or a validated billing algorithm. Practices were defined as billing groups: 3 or more primary care physicians who submit joint billings to OHIP for reimbursement. We excluded practices with fewer than 3 physicians for privacy reasons and those with missing demographic variables. Practices with fewer than 30 patients who met recommendation eligibility were excluded from analyses for that measure to avoid practices with small numbers of cases unduly influencing the practice analyses.

Table 1 describes the inclusion and exclusion criteria, follow- up period and outcome definitions for each measure (Appendix 1 [available at [www.cmajopen.ca/content/5/1/E45/suppl/DC1](http://www.cmajopen.ca/content/5/1/E45/suppl/DC1)] contains full cohort definitions). 

View this table:
[Table 1:](http://www.cmajopen.ca/content/5/1/E45/T1)

Table 1:   Cohort selection criteria

### Outcomes

Our primary outcome was receipt of a potentially low-value test, which we assessed using OHIP claims. Any cohort members with a billing code for a potentially low-value test, as outlined by the recommendations, were considered to have an outcome.

### Statistical analyses

We calculated event rates for each measure for all of Ontario, as well as at the regional and practice levels. Poisson regression was used to investigate temporal trends among each recommendation; when overdispersion was detected, a negative binomial distribution was used instead to obtain more precise estimates of standard error. Regional variation was assessed using the coefficient of variation (across Ontario's 14 Local Health Integration Networks (LHINs) - geographically organized administrative regions that plan, integrate and fund local health care. We assessed variation in ordering across primary care practices by comparing unadjusted event rates for each measure and calculating the coefficient of variation. We considered recommendations that warranted further investigation to be those with substantial room for improvement, defined as having a rate of use of 15% or more, which would better lend itself to quality improvement initiatives designed to reduce frequency of these low-value tests. Recommendations that met these criteria were explored by identifying potential patient-level predictors of testing using 2 sample *t* tests and χ2 tests, where appropriate. All statistical analyses were performed using SAS version 9.4 (SAS Institute).

### Ethics approval

Research ethics approval was received from Sunnybrook Health Sciences Centre, Toronto, Ontario, Canada.

## Results

### Cohort

Between fiscal years 2008/09 and 2012/13, we selected a unique cohort of patients who met eligibility criteria for each Choosing Wisely recommendation: 271 588 patients with low back pain, 2 229 113 patients with an index DEXA scan, and 7 417 444 women aged less than 21 or more than 69 years (Appendix 2, available at [www.cmajopen.ca/content/5/1/E45/suppl/DC1](http://www.cmajopen.ca/content/5/1/E45/suppl/DC1), contains diagrams for each cohort and Appendix 3, available at [www.cmajopen.ca/content/5/1/E45/suppl/DC1](http://www.cmajopen.ca/content/5/1/E45/suppl/DC1), describes patient characteristics for each cohort in 2012/13).

### Provincial frequency of Choosing Wisely recommendations

The most prevalent potentially low-value service was repeated DEXA scanning - 21.0% of patients with an index scan received a second scan within 2 years. Overall, 8.0% of women in the cervical cancer screening population received at least 1 low-value Pap test, with significantly higher screening rates for women less than 21 years of age than those more than 69 years of age (10.8% v. 5.0%; *p* < 0.001). Imaging for low back pain was the least prevalent service, with 4.5% of patients receiving computed tomography or magnetic resonance imaging within 3 months of initial diagnosis. During the study period, the rates of cervical cancer screening decreased significantly (*p* = 0.002), whereas there was no significant trend in rates of imaging for low back pain (*p* = 0.071) or repeat DEXA scans (*p* = 0.17) (Figure 1).

![Figure 1](http://www.cmajopen.ca/https://www.cmajopen.ca/content/cmajo/5/1/E45/F1.medium.gif)

[Figure 1](http://www.cmajopen.ca/content/5/1/E45/F1)

Figure 1 
Annual rates of use of low-value services during the study period (A), annual rates of repeat dual-energy X-ray absorptiometry (DEXA) scans by level of risk during the study period (B) and rates of repeat DEXA scans by risk level of index scan during all study years (C).

### Variability

Substantial variation in rates of use was seen across LHINs (Figure 2). At the regional level, the highest degree of variation was for low back pain imaging (coefficient of variation = 0.35), which ranged from 2.5% to 8.3% across LHINs. The rate of repeat DEXA scans ranged from 13.8% to 28.2% (coefficient of variation = 0.24). The lowest degree of variation was observed for cervical cancer screening (coefficient of variation = 0.14), which ranged from 6.6% to 10.4%.

![Figure 2](http://www.cmajopen.ca/https://www.cmajopen.ca/content/cmajo/5/1/E45/F2.medium.gif)

[Figure 2](http://www.cmajopen.ca/content/5/1/E45/F2)

Figure 2 
**Figure 2:** Regional variation in ordering of CW recommendations by Local Health Integration Network. Note: CW = Choosing Wisely; DEXA = dual energy X-ray absorptiometry.

Figure 3 shows the degree of practice-level variation in the use of low-value testing. Variation was highest for imaging for low back pain (range 0.8%-32.6%; coefficient of variation = 0.59), followed by repeat DEXA scans (range 4.0%-54.9%; coefficient of variation = 0.39) and cervical cancer screening (range 0.9%-35.2%; coefficient of variation = 0.36).

![Figure 3](http://www.cmajopen.ca/https://www.cmajopen.ca/content/cmajo/5/1/E45/F3.medium.gif)

[Figure 3](http://www.cmajopen.ca/content/5/1/E45/F3)

Figure 3 
Practice-level variation in rates of (A) imaging for low back pain, (B) cervical cancer screening and (C) dual-energy X-ray absoptiometry. Each dot represents the rate of use at an individual primary care practice, and the dashed lines represent the mean rate across all practices. 

### Potential predictors of low-value testing

Repeat DEXA scans were the only potential low-value tests with substantial room for improvement and, as such, were the only low-value tests that warranted further investigation into the potential drivers of overuse. Patients who received a repeat scan were significantly older (*p* < 0.001) than those who did not, and women were more likely than men to receive a repeat scan (*p* < 0.001; Table 2). Living in urban or higher income areas was associated with increased testing rates. There were significant differences in rates of use across index scan risk levels: patients who had a high-risk index scan had a much higher repeat testing rate (28.1%) than patients who had a baseline (8.7%) or low-risk (8.1%) scan.

View this table:
[Table 2:](http://www.cmajopen.ca/content/5/1/E45/T2)

Table 2:   Bivariate associations between patient characteristics and having a repeat DEXA scan within 2 years of a previous scan

## Interpretation

The results from this large, retrospective cohort study show that some low-value tests are more common than others, with rates of use ranging from 4.5% for low back pain imaging to 21.0% for repeat DEXA scans. We found substantial variability in rates of use regardless of the frequency of ordering overall, including a 14-fold difference in DEXA ordering and a 40-fold difference in low back pain imaging across practices. Our findings suggest some potential predictors of ordering low-value DEXA scans, including female sex, higher neighbourhood income quintile and the risk level of the index scan.

Although the focus of quality improvement initiatives has predominantly been on misuse and underuse,1 there is emerging research on overuse that offers comparisons for our findings. Among United States Medicare beneficiaries aged 65 years and older, similar frequencies of imaging for low back pain (4.1%-9.4%) and use of cervical cancer screening (6.4%-6.9%) have been reported, although rates of repeat DEXA scans among patients with osteoporosis were lower than in our study (0.8%-1.0%).17 However, a recent study that examined temporal ordering trends of Choosing Wisely recommendations in a large US commercial health plan found substantially higher rates of imaging for low back pain without red flags (53.7%).19 This discrepancy is likely related to varying definitions of low-value care, but may also be influenced by differences in testing practices and populations. Although previous analyses have examined population-level data and regional variation, few studies have assessed rates of use at the level of primary care practices and variability in ordering across practices. Furthermore, our study includes an entire population from a single payer, publicly funded system with no patient copayments, which is novel because most previous research on overuse has been done in the US.16,17,19The lower rates of cervical screening and imaging for low back pain compared with repeat DEXA scans may reflect increased penetrance of guideline recommendations and policy changes. The Ontario government amended the fee schedule to align with the guideline changes made in 2012:20 effective January 2013, cervical cancer screening was not insured for women less than 21 or more than 70 years of age.21 Imaging for low back pain has been the subject of significant quality improvement efforts funded in part by the provincial government.22 Although it is too early to assess the longer term impact of these policy changes on ordering patterns, the likelihood of further improvements in population rates for cervical screening and low back pain imaging seems low.

### Limitations

Administrative databases do not provide important clinical information, such as presence of symptoms or abnormal findings on physical examination that are necessary in determining appropriateness of ordering tests.23 For example, clinical information not captured may identify patients with an accelerated expected rate of bone loss for whom repeat scans may be appropriate. 

Our rate of imaging for low back pain may be underestimated. Because administrative data lack clinical nuance, we developed an exhaustive list of red flag exclusions for this cohort to avoid capturing false-positives; however, excluding higher risk patients may have biased our sample. Earlier literature has shown that the rate of low back pain scans for red flag conditions is a minority of overall ordering.24

We were unable to measure individual physician ordering behaviour within a practice, and thus are unable to assess the extent to which an individual physician may skew the results of the practice. In addition, we cannot determine whether the ordering physician for the test was the primary care doctor or a subspecialist, although previous research has shown that most ordering is done by a family physician.25 Furthermore, we were unable to account for factors that may account for differences in ordering across practices, including decision support tools associated with an electronic medical record.26

Finally, as we conducted exploratory analyses into potential predictors, we were unable to assess the adjusted association of patient, provider, and practice characteristics with low-value DEXA testing. Provider-level factors, including years since graduation or practice setting, would be valuable in identifying potential intervention targets. However, our findings identify potential predictors that future research may build upon.

### Conclusion

In this large, population-based, retrospective study of low-value services in primary care, we found significant regional and practice-level variation for all 3 services studied. Although rates of cervical screening and imaging for low back pain were lower than expected, significant practice variation was seen, and low-value DEXA scans were common. The results suggest opportunities for further reducing low-value care in primary care.

Our results have considerable importance for public policy. The dramatic variation in ordering across practices in a large jurisdiction like Ontario suggests more work is needed to understand the drivers of low-value care at the patient, practice and provider levels.

### Supplemental information

For reviewer comments and the original submission of this manuscript, please see [www.cmajopen.ca/content/5/1/E45/suppl/DC1](http://www.cmajopen.ca/content/5/1/E45/suppl/DC1)

## Acknowledgements

Disclaimer: Parts of this material are based on data and information compiled and provided by the Canadian Institute for Health Information (CIHI). However, the analyses, conclusions, opinions and statements expressed herein are those of the author, and not necessarily those of CIHI. Parts of this material are based on data and information provided by Cancer Care Ontario. The opinions, results, view, and conclusions reported in this paper are those of the authors and do not necessarily reflect those of Cancer Care Ontario. No endorsement by Cancer Care Ontario is intended or should be inferred.

## Footnotes

*   **Competing interests:** None declared.

*   **Contributors:** Ciara Pendrith and Sacha Bhatia were responsible for the study concept and design. Ciara Pendrith, Graham Mecredy and Sacha Bhatia acquired the data. Ciara Pendrith and Graham Mecredy did the statistical analyses. All authors analyzed and interpreted the data. Ciara Pendrith, Meghan Bhatia and Sacha Bhatia drafted the manuscript. All authors critically revised the manuscript for important intellectual content, approved the final version to be published and agreed to act as guarantors of the work.

*   **Funding:** This study was supported by the Institute for Clinical Evaluative Sciences (ICES), which is funded in part by an annual grant from the Ontario Ministry of Health and Long-Term Care (MOHLTC). The funders had no role in the design and conduct of the study; the collection, management, analysis or interpretation of the data; the preparation, review or approval of the manuscript; or the decision to submit the manuscript for publication. The opinions, results and conclusions reported in this paper are those of the authors and are independent from the funding sources. No endorsement by ICES or the Ontario MOHLTC is intended or should be inferred. Noah Ivers is supported by New Investigator Awards from the Canadian Institutes of Health Research and the Department of Family and Community Medicine, University of Toronto.

## References

1.  Institute of Medicine (2001) Crossing the quality chasm: a new health system for the 21st century. Washington (DC): National Academy Press.
    
    

2.  Wennberg JE, Fisher ES, Skinner JS (2002) Geography and the debate over Medicare reform. Health Aff (Millwood) (Suppl Web Exclusives):W96–114.
    
    

3.  Levinson W, Kallewaard M, Bhatia RS, et al., Choosing Wisely International Working Group (2015) 'Choosing Wisely': a growing international campaign. BMJ Qual Saf 24:167–74.
    
    [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjI0LzIvMTY3IjtzOjQ6ImF0b20iO3M6MTk6Ii9jbWFqby81LzEvRTQ1LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

4.  Wolfson D, Santa J, Slass L (2014) Engaging physicians and consumers in conversations about treatment overuse and waste: a short history of the choosing wisely campaign. Acad Med 89:990–5.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1097/ACM.0000000000000270&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=24979166&link_type=MED&atom=%2Fcmajo%2F5%2F1%2FE45.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000338210000017&link_type=ISI) 

5.  Levinson W, Huynh T (2014) Engaging physicians and patients in conversations about unnecessary tests and procedures: Choosing Wisely Canada. CMAJ 186:325–6.
    
    [FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czo5OiIxODYvNS8zMjUiO3M6NDoiYXRvbSI7czoxOToiL2NtYWpvLzUvMS9FNDUuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

6.  (2014) Choosing Wisely Canada family medicine: eleven things physicians and patients should question. Ottawa: Canadian Medical Association/Toronto: University of Toronto. Available[www.choosingwiselycanada.org/recommendations/family-medicine/](http://www.choosingwiselycanada.org/recommendations/family-medicine/). accessed 2015 Feb. 6.
    
    

7.  Bhatia RS, Levinson W, Shortt S, et al. (2015) Measuring the effect of Choosing Wisely: an integrated framework to assess campaign impact on low-value care. BMJ Qual Saf 24:523–31.
    
    [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjI0LzgvNTIzIjtzOjQ6ImF0b20iO3M6MTk6Ii9jbWFqby81LzEvRTQ1LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

8.  American Academy of Familty Physicians (2013) Fifteen things physicians and patients should question. In: Choosing Wisely: an initiative of the ABIM Foundation. Philadelphia: ABIM Foundation. Available[www.choosingwisely.org/societies/american-academy-of-family-physicians/](http://www.choosingwisely.org/societies/american-academy-of-family-physicians/). accessed 2015 Nov. 23.
    
    

9.  (2014) Choosing Wisely Canada radiology: five things physicians and patients should question. Ottawa: Canadian Medical Association/Toronto: University of Toronto. Available[www.choosingwiselycanada.org/recommendations/radiology/](http://www.choosingwiselycanada.org/recommendations/radiology/). accessed 2015 Feb. 6.
    
    

10. (2014) Choosing Wisely Canada pathology: five things physicians and patients should question. Ottawa: Canadian Medical Association/Toronto: University of Toronto. Available[www.choosingwiselycanada.org/recommendations/pathology/](http://www.choosingwiselycanada.org/recommendations/pathology/). accessed 2015 Feb. 6.
    
    

11. American College of Rheumatology (2013) Five things physicians and patients should question. In: Choosing Wisely: an initiative of the ABIM Foundation. Philadelphia: ABIM Foundation. Available[www.choosingwisely.org/societies/american-college-of-rheumatology/](http://www.choosingwisely.org/societies/american-college-of-rheumatology/). accessed 2015 Nov. 23.
    
    

12. Chow SL, Carter Thorne J, Bell MJ, et al., Canadian Rheumatology Association Choosing Wisely Committee (2015) Choosing wisely: the Canadian Rheumatology Association's list of 5 Items physicians and patients should question. J Rheumatol 42:682–9.
    
    [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NjoianJoZXVtIjtzOjU6InJlc2lkIjtzOjg6IjQyLzQvNjgyIjtzOjQ6ImF0b20iO3M6MTk6Ii9jbWFqby81LzEvRTQ1LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

13. Iron K, Jaakimainen L, Rothwell D, et al. (2004) Investigation of acute lower back pain in Ontario: are guidelines being followed? Toronto: Institute for Clinical Evaluative (ICES).
    
    

14. Jaglal S, Hawker G, Croxford R, et al. (2014) Impact of a change in physician reimbursement on bone mineral density testing in Ontario, Canada: a population-based study. CMAJ Open 2:E45–50.
    
    

15. Lofters AK, Moineddin R, Hwang SW, et al. (2010) Low rates of cervical cancer screening among urban immigrants: a population-based study in Ontario, Canada. Med Care 48:611–8.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1097/MLR.0b013e3181d6886f&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=20548258&link_type=MED&atom=%2Fcmajo%2F5%2F1%2FE45.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000279428200008&link_type=ISI) 

16. Colla CH, Morden NE, Sequist TD, et al. (2015) Choosing wisely: prevalence and correlates of low-value health care services in the United States. J Gen Intern Med 30:221–8.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1007/s11606-014-3070-z&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=25373832&link_type=MED&atom=%2Fcmajo%2F5%2F1%2FE45.atom) 

17. Schwartz AL, Landon BE, Elshaug AG, et al. (2014) Measuring low-value care in Medicare. JAMA Intern Med 174:1067–76.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1001/jamainternmed.2014.154114&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=24819824&link_type=MED&atom=%2Fcmajo%2F5%2F1%2FE45.atom) 

18. Allin S, Munce S, Jaglal S, et al. (2014) Capture of osteoporosis and fracture information in an electronic medical record database from primary care. AMIA Annu Symp Proc 2014:240–8.
    
    

19. Rosenberg A, Agiro A (2015) Gottlieb Met al. Early trends among seven recommendations from the Choosing Wisely campaign. JAMA Intern Med 175:1913–20.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1001/jamainternmed.2015.5441&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=26457643&link_type=MED&atom=%2Fcmajo%2F5%2F1%2FE45.atom) 

20. Murphy J, Kennedy EB, Dunn S, et al. (2012) Ontario Cervical Screening Program; Program in Evidence-based Care. Cervical screening: a guideline for clinical practice in Ontario. J Obstet Gynaecol Can 34:453–8.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/S1701-2163(16)35242-2&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=22555138&link_type=MED&atom=%2Fcmajo%2F5%2F1%2FE45.atom) 

21. (2013) Implementation of 2012 Physicians Services Agreement - Amendments to the schedule of benefits for physician service - Effective January 1, 2013. INFOBulletin no 4585. North York (ON): Ontario Ministry of Health and Long-Term Care. Available[www.health.gov.on.ca/en/pro/programs/ohip/bulletins/4000/bul4585.pdf](http://www.health.gov.on.ca/en/pro/programs/ohip/bulletins/4000/bul4585.pdf). accessed 2015 Nov. 23.
    
    

22. (2012) Provincial strategy for x-ray computed tomography (CT) and/or magnetic resonance imaging (MRI) for low back pain. INFOBulletin No 4569. North York (ON): Ontario Ministry of Health and Long-Term Care. Available[www.health.gov.on.ca/en/pro/programs/ohip/bulletins/4000/bul4569.pdf](http://www.health.gov.on.ca/en/pro/programs/ohip/bulletins/4000/bul4569.pdf). accessed 2015 Nov. 23.
    
    

23. Hawker GA, Wright JG, Coyte PC, et al. (2001) Determining the need for hip and knee arthroplasty: the role of clinical severity and patients' preferences. Med Care 39:206–16.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1097/00005650-200103000-00002&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=11242316&link_type=MED&atom=%2Fcmajo%2F5%2F1%2FE45.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000167279400002&link_type=ISI) 

24. Gidwani R, Sinnott P, Avoundjian T, et al. (2016) Inappropriate ordering of lumbar spine magnetic resonance imaging: are providers choosing wisely? Am J Manag Care 22:e68–76.
    
    

25. Jaglal SB, McIsaac WJ, Hawker G, et al. (2000) Patterns of use of the bone mineral density test in Ontario, 1992-1998. CMAJ 163:1139–43.
    
    [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czoxMDoiMTYzLzkvMTEzOSI7czo0OiJhdG9tIjtzOjE5OiIvY21ham8vNS8xL0U0NS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

26. Kaushal R, Shojania KG, Bates DW (2003) Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med 163:1409–16.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1001/archinte.163.12.1409&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=12824090&link_type=MED&atom=%2Fcmajo%2F5%2F1%2FE45.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000183705400005&link_type=ISI) 

*   Copyright 2017, Joule Inc. or its licensors