Use of patient-reported outcomes in regional cancer centres over time: a retrospective study
============================================================================================

* Lisa Barbera
* Faith Lee
* Rinku Sutradhar

## Abstract

**Background:** Since 2007, Cancer Care Ontario has been collecting data using the Edmonton Symptom Assessment System as a patient-reported outcome measure for use in routine care. The purpose of this project was to evaluate the factors associated with Edmonton Symptom Assessment System uptake among cancer patients seen at regional cancer centres and to examine if these associations have changed over time.

**Methods:** This was a retrospective cohort study among cancer patients eligible to complete Edmonton Symptom Assessment System assessments who were seen at regional cancer centres in Ontario between 2007 and 2015. We used linked administrative sources of health care data. Our primary outcome for each patient was defined as the rate of ESAS assessments, which was analyzed overall and on an annual basis.

**Results:** We identified 525 409 unique patients with at least 1 visit to a cancer centre during the study period. The percentage of patients with at least 1 Edmonton Symptom Assessment System assessment increased from 5% in 2007 to 67% in 2015. Analysis demonstrated that variation by region and by cancer type decreased over time: relative rates for region ranged from 0.31 to 13.3 in 2007 whereas they ranged from 0.7 to 1.56 in 2015, and relative rates for cancer type ranged from 0.03 to 1.0 in 2007 whereas they ranged from 0.55 to 1.0 in 2015. In 2015 women and people living in poorer neighbourhoods had a lower Edmonton Symptom Assessment System uptake (relative rate 0.93 and 0.91, respectively).

**Interpretation:** Ontario has implemented a patient-reported outcome program across the province; over time, uptake has improved and variation by cancer type and region has decreased. Variation persists for other patient characteristics, which suggests that there are opportunities to improve equity in the program.

Patient-reported outcome measures are tools or instruments used to capture patients’ health status from their perspective.1,2 These measures have been used for some time in research.3 Increasingly, they are being incorporated into routine clinical care.4,5 There is evidence that they improve symptom identification, symptom monitoring over time, communication and quality of life.6–9 There is emerging evidence that their routine use may decrease emergency department visits and even improve survival.10–13

In 2007, Cancer Care Ontario implemented a province-wide program to screen for common cancer symptoms using the Edmonton Symptom Assessment System (ESAS). Other jurisdictions have also ventured into this space. For example, there is a large program that gathers patient-reported outcomes in the Netherlands for pediatrics.14 The Dartmouth–Hitchcock Medical Center has an institution-wide program as another example.15 In the United Kingdom the National Health Service routinely collects patient-reported outcomes for orthopedic patients and it is expanding this practice to other patient populations.16 The Swedish National Quality Registers have also started to collect patient-reported outcomes.17 The program at Cancer Care Ontario, however, is one of the largest, most comprehensive patient-reported outcome programs in existence.

The purpose of this project was to evaluate the rate of ESAS use over time among cancer patients seen at regional cancer centres. We also examined the factors associated with ESAS use and if these associations have changed over time. Cancer Care Ontario has made significant efforts to support the use of ESAS in clinics, but the change has been slow. This paper aims to provide a robust description of the changes in ESAS use over time and by region and to evaluate how other patient, tumour and system factors might be associated with ESAS uptake over time.

## Methods

### Study design and setting

We conducted a retrospective cohort study among ESAS-eligible cancer patients in Ontario, Canada. The study used administrative sources of health care data linked via a unique encoded identifier. It was not possible to approach this question by conducting a formal implementation evaluation. The implementation of the ESAS program is an ongoing effort at 14 cancer centres that cumulatively reflects a myriad of local efforts. It would not be possible to catalogue these efforts over time or attribute changes in ESAS rates to any 1 particular endeavour. ESAS implementation went live in all centres in 2007 for patients with lung cancer and patients attending palliative care clinics. This was a central strategy direction from Cancer Care Ontario. Some centres were able to mobilize and act quickly but others struggled. By 2010 most regional cancer centres had expanded implementation to include all types of cancer. How this expansion occurred was left to local sites. Precise documentation of these changes is beyond the scope of this project and efforts to do this are unlikely to be successful given staff changes over the past decade.

In Ontario, radiation treatment and a substantial proportion of systemic therapy are provided in regional cancer centres. Some systemic therapy is provided at partner hospitals. Some patients with cancer (especially those treated only with surgery) are never seen at a cancer centre. The provincial patient-reported outcomes program that oversees ESAS implementation is active in all regional cancer centres. While implementation is also active at some partner hospitals it is not consistent, and these facilities are not included in this study.

Every 5 years Cancer Care Ontario publishes a provincial cancer plan that outlines strategic priorities for the organization. Although ESAS was introduced in 2007, the most recent cancer plan was the first one to specifically identify patient-reported outcomes, reflecting the increasing importance of patient-reported outcomes within the organization. To facilitate the implementation of the collection of patient-reported outcomes, Cancer Care Ontario built and maintains a Web-based platform to administer patient-reported outcome measures in the clinic. All symptom reports are collected centrally in the Symptom Management Reporting Database. It also provides support to each centre for ongoing implementation, sustainability and quality improvement work via service agreements. In spite of the central mandate, each centre has implemented patient-reported outcomes in ways that suit the local context. Over time, cancer centres have been monitored for their performance on how many patients are screened each month.

ESAS is a 9-item instrument that asks patients to rate the intensity of their symptoms on a scale of 0–10.18 The 9 symptoms included are anxiety, depression, fatigue, drowsiness, pain, shortness of breath, nausea, appetite and well-being. It was originally developed for use with patients with cancer who were receiving palliative care but has since been validated in general oncology patients.19 Implementation of ESAS started in a limited group of patients, but by 2010 it had been rolled out more broadly.20–22 As of the time of writing, every cancer patient attending a regional cancer centre is encouraged to complete ESAS at a kiosk in the waiting room before being seen by their medical team. The output is intended to be used in the clinical encounter and to facilitate a discussion about symptoms and care.

### Study population and observation window

ESAS-eligible cancer patients were adults (> 18 yr) who visited any of the regional cancer centres in Ontario between Apr. 1, 2007, and Dec. 31, 2015. These patients were identified by the presence of records in the Activity Level Reporting database. Eligible visits were visits to any of the regional cancer centre programs (e.g., radiation program, systemic program) except preventive oncology or research, as ESAS may not be used at those types of visits. Exclusion criteria for our cohort were defined as having 1 or more of the following: invalid unique encoded identifier, missing date of birth, non-first cancer diagnosis, or death date before cohort entry. People with invalid visit program codes in the Activity Level Reporting database were also excluded. Subsequently, if people in our cohort had missing information on the covariates of interest, they were excluded as well. Patients were followed until Dec. 31, 2015, subsequent cancer diagnosis or date of death listed in the Registered Persons Database, whichever occurred first.

### Outcome definition

Our primary outcome for each patient was defined as the rate of ESAS assessments, calculated overall and annually. Patients were assigned to annual cohorts provided they had a clinic visit record in the Activity Level Reporting database during that year. This is aligned with Cancer Care Ontario’s measure of symptom screening activity, which is also person based. This database contains records of visits and services occurring at each cancer centre. It is mandatory for each centre to report their volumes of clinical service to Cancer Care Ontario via this database. As such, in any given year, all patients being analyzed are unique. However, over multiple years (that is, over multiple annual cohorts), patients are not necessarily unique from year to year. People seen at a cancer centre in multiple years will be counted in each year that they have a visit. Within that year, even if they have multiple ESAS assessments completed, they are counted only once. In any given year, the rate was calculated as the number of ESAS assessments divided by the total follow-up time in that year. This measurement approach allows us to adequately evaluate changes in the screening rate over time. For descriptive purposes, we also examined ESAS uptake as a binary outcome; if patients had not undertaken any ESAS assessment in that year they were classified as non-ESAS users, otherwise they were classified as ESAS users. Dates for ESAS assessments were determined from the ESAS database.

### Covariate definitions and data sources

Patient, tumour and system variables were chosen to adjust for possible confounders when evaluating ESAS rate. Age was retrieved from the Registered Persons Database, which contains sociodemographic information for all Ontario Health Insurance Plan beneficiaries.23 The type of cancer diagnosis was determined from the Ontario Cancer Registry.24,25 Neighbourhood income quintile at the start of each year was determined by linking postal codes and residential codes in the Registered Persons Database to census data.26 Region of residence was similarly determined. Cancer Care Ontario’s public reporting has long identified variation by region ([www.csqi.on.ca](http://www.csqi.on.ca)). Given that it would not be reasonable to place both local health integration network (LHIN) and cancer centre into the model and region can function as a surrogate for cancer centre, we elected to incorporate only region into the regression model. All Cancer Care Ontario reporting is by region. Charlson score27 was determined on the basis of records from the Canadian Institute for Health Information’s Discharge Abstract Database and Same Day Surgery Database. These data sets document diagnoses coded at hospital admissions.28 The score was calculated with a 2-year look-back window. Comorbidity was also assessed using Aggregated Diagnosis Groups on the basis of a 2-year look-back period, founded on the Johns Hopkins Adjusted Clinical Group System.29 Each patient could be assigned anywhere from 0 to 32 Aggregated Diagnosis Groups. For this study, the Aggregated Diagnosis Groups was regrouped into 3 categories: ≥ 10, 6–10 and 0–5.30 Each patient’s mean resource intensity weight was measured using the resource utilization band on the basis of a 2-year look-back window. For the purpose of our analyses, the resource utilization band scores were analyzed similarly to the way they were analyzed in prior work.31 Patients were assigned to 1 of 6 resource utilization band categories, where 0 implied that the patient was a non-user and 5 was the highest level of resource use. Multiple variables for comorbidity and/or resource use were included in the model to adjust for possible confounding between illness level and the likelihood of having an ESAS assessement (e.g., sicker patients might be more likely to visit the cancer centre and therefore complete an ESAS assessment). This would facilitate a reasonable comparison among regions. A patient was identified as an immigrant if there was a “date of landing” record in Immigration, Refugees and Citizenship Canada’s Permanent Resident Database.32 This data set provides demographic information for all legal immigrants to Canada including country of birth, citizenship, country of last permanent residence and date of immigration. We further identified immigrants as either recent (< 5 yr since immigration) or long-term (≥ 5 yr since immigration).

### Statistical analysis

We analyzed the overall cohort, which consisted of all unique patients accrued over the study period, and we subsequently analyzed each annual cohort of patients. The distributions of the baseline characteristics of the overall cohort and the distributions of the baseline characteristics for each of the annual cohorts were assessed. Counts and proportions were used to describe the categorical variables; mean, median and interquartile range (IQR) were used to describe the continuous variables. As preliminary work, histograms were developed for each year to illustrate the distribution of the number of ESAS assessments among those who had at least 1 ESAS assessment in that year. In addition, the proportion of patients who had at least 1 ESAS assessment in that year was overlaid on the histogram as a horizontal line.

Factors associated with the rate of ESAS uptake were first examined in our overall cohort. As the number of ESAS assessments along with follow-up time varies significantly across patients, a negative binomial regression model was implemented. The natural logarithm of the follow-up time was used as an offset term in the model. A generalized estimating equations (GEE) approach with an exchangeable correlation structure was imposed to account for possible correlation that may have arisen because of the annual repeated measures on each patient.33 Characteristics included in the model were age, sex, income quintile, immigration status, region of residence, cancer type, Charlson group, Aggregated Diagnosis Group and resource utilization band group. Both univariable and multivariable regression models were implemented. Collinearity between the variables was assessed using the variance inflation factor, where a cut-off of 5 or higher was used as an indication of collinearity.

Factors associated with the rate of ESAS uptake were also examined on an annual basis, to determine if the associations were changing over time. Because there were no repeated observations from the same patient within any given year, we used a negative binomial regression model (without a GEE approach) and conducted both univariable and multivariable analyses. All analyses were completed using SAS version 9.3 and R statistical software version 3.2.3.

### Ethics approval

The study was conducted in accordance with the strict privacy and confidentiality policies of ICES. It involved only secondary data analyses and was exempt from requiring approval from a research etchics board.

## Results

We identified 525 409 unique patients with at least 1 visit to a cancer centre between 2007 and 2015. A total of 5908 patients were excluded because of missing covariate information (*n* = 3235) or invalid visit type (*n* = 2673). The group with missing information constituted about 1% of the cohort across all levels of covariates. This means that some characteristics had far less than 1% missing and thus could not be described because of reporting constraints associated with small sample sizes. Since there was no reason to suspect a pattern for the missing information and since this small percentage would not influence our final interpretations, we have not provided further descriptions of this group. Cohort characteristics are presented in Table 1.

View this table:
[Table 1:](http://www.cmajopen.ca/content/7/1/E101/T1)

Table 1: 
Cohort characteristics at baseline

Appendix 1 (available at [www.cmajopen.ca/content/7/1/E101/suppl/DC1](http://www.cmajopen.ca/content/7/1/E101/suppl/DC1)) shows the uptake of ESAS in alternating years from 2007 to 2015 (for simplicity, not every year is shown). The proportion of patients with at least 1 ESAS assessment increased from 5% in 2007 to 67% in 2015 (represented by the horizontal dotted line, right-sided y-axis). This appendix also demonstrates the distribution of the number of ESAS assessments among responders (left-sided y-axis). Patients most commonly had 1 or 2 assessments per year.

The negative binomial GEE model for examining factors associated with the rate of ESAS uptake, using the entire cohort (*n* = 525 409), indicated that the relative rates from both univariable and multivariable models were similar except for sex. The estimate from the univariable model showed that women had a 10% higher ESAS uptake rate than men. However, after multivariable adjustment, the rate of ESAS uptake was 5% lower among women than among men. With lung cancer as the reference cancer type, all other cancer types were associated with lower ESAS use. With region 7 as the reference level, all other regions were associated with higher ESAS use (Figure 1).

![Figure 1:](http://www.cmajopen.ca/https://www.cmajopen.ca/content/cmajo/7/1/E101/F1.medium.gif)

[Figure 1:](http://www.cmajopen.ca/content/7/1/E101/F1)

Figure 1: 
Negative binomical generalized estimating equations (GEE) output. Note: ADG = Aggregated Diagnosis Group, CI = confidence interval, RUB = resource utilization band. The local health integration networks (LHINs) are as follows: LHIN 1 = Erie St. Clair, LHIN 2 = South West, LHIN 3 = Waterloo Wellington, LHIN 4 = Hamilton Niagara Haldimand Brant, LHIN 5 = Central West, LHIN 6 = Mississauga Halton, LHIN 7 = Toronto Central, LHIN 8 = Central, LHIN 9 = Central East, LHIN 10 = South East, LHIN 11 = Champlain, LHIN 12 = North Simcoe Muskoka, LHIN 13 = North East, LHIN 14 = North West.

Appendix 2 (available at [www.cmajopen.ca/content/7/1/E101/suppl/DC1](http://www.cmajopen.ca/content/7/1/E101/suppl/DC1)) describes the characteristics of the annual cohorts in 2007, 2011 and 2015 (for simplicity, not every year is shown). The distribution of characteristics was similar from year to year, with the exception of immigration status and Charlson score.

The forest plots shown in Figures 2 and 3 illustrate the results from the negative binomial multivariable regression model for the years 2007, 2011 and 2015. Figure 2 displays the forest plots for cancer type and comorbidity. In 2007, the relative rates of ESAS uptake by cancer type were much less than 1 across all cancer types, using lung cancer as a reference, with a relative rate as low as 0.031 for prostate cancer. By 2015 this range had improved considerably, although prostate cancer still had the lowest relative rate.

![Figure 2:](http://www.cmajopen.ca/https://www.cmajopen.ca/content/cmajo/7/1/E101/F2.medium.gif)

[Figure 2:](http://www.cmajopen.ca/content/7/1/E101/F2)

Figure 2: 
Relative rates of Edmonton Symptom Assessment System (ESAS) uptake from multivariable regression model for cancer type and comorbidity. Note: ADG = Aggregated Diagnosis Group, CI = confidence interval, RUB = resource utilization band.

![Figure 3:](http://www.cmajopen.ca/https://www.cmajopen.ca/content/cmajo/7/1/E101/F3.medium.gif)

[Figure 3:](http://www.cmajopen.ca/content/7/1/E101/F3)

Figure 3: 
Relative rates of Edmonton Symptom Assessment System (ESAS) uptake from multivariable regression model for age, sex, income quintile, immigration status and region (local health integration network; LHIN). Note: CI = confidence interval, Immi = Immigration, IQ = income quintile. The LHINs are as follows: LHIN 1 = Erie St. Clair, LHIN 2 = South West, LHIN 3 = Waterloo Wellington, LHIN 4 = Hamilton Niagara Haldimand Brant, LHIN 5 = Central West, LHIN 6 = Mississauga Halton, LHIN 7 = Toronto Central, LHIN 8 = Central, LHIN 9 = Central East, LHIN 10 = South East, LHIN 11 = Champlain, LHIN 12 = North Simcoe Muskoka, LHIN 13 = North East, LHIN 14 = North West.

Figure 3 displays the forest plots for the remaining characteristics. Age had a consistent relative rate across the years. Sex did not have a significant effect in 2007 but it did in 2011 and 2015. In 2007 there was no significant association between income quintile and ESAS uptake. In 2015, income quintiles 1 and 2 (poorer) were associated with lower ESAS uptake than income quintile 5 (richer). ESAS uptake by long-term immigrants did not differ significantly from that by non-immigrants in 2007 but it was lower in the former group in 2011 and 2015. On the other hand, recent immigrants were found to have a 34% fewer ESAS assessments than long-term immigrants in 2007, but the number of assessments between these two groups did not differ significantly in 2011 and 2015. In terms of the relative rates of ESAS assessments by region, we observed that the largest variation was in 2007, where the relative rates ranged from 0.31 (95% confidence interval [CI] 0.26–0.38) to 13.3 (95% CI 11.65–15.20). By 2015, the range had decreased, from 0.7 (95% CI 0.67–0.73) to 1.56 (95% CI 1.51–1.61).

## Interpretation

We have shown that ESAS uptake in Ontario cancer centres has increased considerably over time. Our results clearly demonstrate the effectiveness of the implementation of a patient-reported outcome program on a large scale. The amount of variation seen in association with certain variables has improved. For example, there was much less variation by region and cancer type in 2015 than in earlier years. Patients with comorbid illnesses are more likely to be screened than those without comorbid illnesses, which mitigates concerns that patients with more complex needs are being missed. However, there is still variation in terms of other variables, raising the possibility of ongoing equity issues.

Cancer Care Ontario’s role as a provincial cancer agency is a key factor in the uptake of this program. Including patient-reported outcomes in the provincial cancer plan makes this a clear strategic priority. Dedicated funding to local centres to support ongoing implementation and execution of the program supports this priority. Cancer Care Ontario monitors several performance measures for each regional program, including the ESAS screening rate. This is evaluated with senior leadership on a quarterly basis.22,34 This performance management system may have contributed to decreasing variation across regions. Other ongoing quality improvement activities such as annual chart reviews to assess symptom management and patient surveys of their experience with ESAS have probably also helped to sustain symptom screening activity.

The implementation has reached all tumour sites, having started primarily with lung cancer. Prostate cancer remains the cancer site screened least often. It has been reported by clinicians that the ESAS items are not always relevant to their patient population.35 In 2016, Cancer Care Ontario began province-wide implementation of the Expanded Prostate Cancer Index-Clinical Practice (EPIC-CP).36,37 This measure has urinary, bowel and sexual function domains that are highly relevant to prostate cancer patients.

The uptake of symptom screening by sex, income and immigration status has changed over time, in some cases improving and in others worsening. It may be that deprived individuals stand to benefit the most from standardized symptom screening. For example, Basch and colleagues reported that those who were computer inexperienced benefited the most from the intervention.10 Equity issues will need to be a focus of ongoing quality improvement efforts locally and programmatic changes provincially.

Strengths of this paper are that we included patients attending regional cancer centres in the denominator, which ensured that they all were eligible to complete ESAS assessments. The data for this population are extensive and population based. The use of ESAS uptake rate as an outcome accommodates for the varying amounts of follow-up time for individual patients. The factors evaluated included patient, tumour and system factors.

### Limitations

Our study has several limitations. More granular details beyond immigration status (such as fluency in English) were not available. Although we were able to observe how frequently ESAS assessments were completed, we were not able to draw conclusions about how the data were actually used in care. The results may not be generalizable to other jurisdictions. The Activity Level Reporting data set has not been validated although reporting is mandatory. The Immigration, Refugees and Citizenship Canada data set has also not been validated. An alternative approach to measuring the outcome might have been to use a visit-based indicator. However, given that all covariates were patient based, not visit based, this alternative outcome definition is unlikely to have changed our conclusions. Furthermore, our measurement approach is more closely aligned with Cancer Care Ontario’s measurement approach, making our observations more directly applicable to the program.

### Conclusion

Patient-reported outcomes are becoming a more common feature of routine clinical care. The cancer care system in Ontario has implemented symptom screening across the system. It is hoped that cancer-specific measures, such as the prostate measure, will further improve clinician and patient engagement with the program. Opportunities remain to improve the uptake of ESAS overall and to decrease inequities in uptake among certain segments of the population.

## Footnotes

*   **Competing interests:** Lisa Barbera is the former provincial lead for patient-reported outcomes at Cancer Care Ontario.

*   This article has been peer reviewed.

*   **Contributors:** Lisa Barbera and Rinku Sutradhar contributed to the conception and design of the study and the acquisition of the data. All authors participated in analyzing and interpreting the data, drafting the manuscript and revising it critically for intellectual content. All authors approved the final version of the manuscript and agreed to be accountable for all aspects of the work.

*   **Funding:** This study was conducted with the support of the Canadian Cancer Society Research Institute.

*   **Disclaimer:** This study was supported by ICES, which is funded by an annual grant from the Ontario Ministry of Health and Long-Term Care (MOHLTC). Parts of this material are based on data and information provided by Cancer Care Ontario. The opinions, results, views and conclusions reported in this article are those of the authors and are independent from the funding sources. No endorsement by ICES, the Ontario MOHLTC or Cancer Care Ontario is intended or should be inferred.

*   **Supplemental information:** For reviewer comments and the original submission of this manuscript, please see [www.cmajopen.ca/content/7/1/E101/suppl/DC1](http://www.cmajopen.ca/content/7/1/E101/suppl/DC1).

## References

1.  (2009) Guidance for industry: patient-reported outcomes measures — use in medical product development to support labeling claims (US Food and Drug Administration, Silver Spring (MD)), 74, 235, pp 65132–3.
    
    

2.  Acquadro C, Berzon R, Dubois D, et al., PRO Harmonization Group (2003) Incorporating the patient’s perspective into drug development and communication: an ad hoc task force report of the Patient-Reported Outcomes (PRO) Harmonization Group meeting at the Food and Drug Administration, February 16, 2001. Value Health 6:522–31.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1046/j.1524-4733.2003.65309.x&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=14627058&link_type=MED&atom=%2Fcmajo%2F7%2F1%2FE101.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000185678300004&link_type=ISI) 

3.  Au HJ, Ringash J, Brundage M, et al., NCIC CTG Quality of Life Committee (2010) Added value of health-related quality of life measurement in cancer clinical trials: the experience of the NCIC CTG. Expert Rev Pharmacoecon Outcomes Res 10:119–28.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1586/erp.10.15&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=20384559&link_type=MED&atom=%2Fcmajo%2F7%2F1%2FE101.atom) 

4.  Snyder CF, Aaronson NK (2009) Use of patient-reported outcomes in clinical practice. Lancet 374:369–70.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/S0140-6736(09)61400-8&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=19647598&link_type=MED&atom=%2Fcmajo%2F7%2F1%2FE101.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000268815200009&link_type=ISI) 

5.  Greenhalgh J, Meadows K (1999) The effectiveness of the use of patient-based measures of health in routine practice in improving the process and outcomes of patient care: a literature review. J Eval Clin Pract 5:401–16.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1046/j.1365-2753.1999.00209.x&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=10579704&link_type=MED&atom=%2Fcmajo%2F7%2F1%2FE101.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000083683400005&link_type=ISI) 

6.  Kotronoulas G, Kearney N, Maguire R, et al. (2014) What is the value of the routine use of patient-reported outcome measures toward improvement of patient outcomes, processes of care, and health service outcomes in cancer care? A systematic review of controlled trials. J Clin Oncol 32:1480–501.
    
    [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiamNvIjtzOjU6InJlc2lkIjtzOjEwOiIzMi8xNC8xNDgwIjtzOjQ6ImF0b20iO3M6MjA6Ii9jbWFqby83LzEvRTEwMS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

7.  Chen J, Ou L, Hollis SJ (2013) A systematic review of the impact of routine collection of patient reported outcome measures on patients, providers and health organisations in an oncologic setting. BMC Health Serv Res 13:211.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1186/1472-6963-13-211&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=23758898&link_type=MED&atom=%2Fcmajo%2F7%2F1%2FE101.atom) 

8.  Yang LY, Manhas DS, Howard AF, et al. (2018) Patient-reported outcome use in oncology: a systematic review of the impact on patient-clinician communication. Support Care Cancer 26:41–60.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmajo%2F7%2F1%2FE101.atom) 

9.  Howell D, Liu G (2011) Can routine collection of patient reported outcome data actually improve person-centered health? Healthc Pap 11:42–7, discussion 55–8.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=22543292&link_type=MED&atom=%2Fcmajo%2F7%2F1%2FE101.atom) 

10. Basch E, Deal AM, Kris MG, et al. (2016) Symptom monitoring with patient-reported outcomes during routine cancer treatment: a randomized controlled trial. J Clin Oncol 34:557–65.
    
    [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiamNvIjtzOjU6InJlc2lkIjtzOjg6IjM0LzYvNTU3IjtzOjQ6ImF0b20iO3M6MjA6Ii9jbWFqby83LzEvRTEwMS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

11. Basch E, Deal AM, Dueck AC, et al. (2017) Overall survival results of a trial assessing patient-reported outcomes for symptom monitoring during routine cancer treatment. JAMA 318:197–8.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1001/jama.2017.7156&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=28586821&link_type=MED&atom=%2Fcmajo%2F7%2F1%2FE101.atom) 

12. Denis F, Lethrosne C, Pourel N, et al. (2017) Randomized trial comparing a web-mediated follow-up with routine surveillance in lung cancer patients. J Natl Cancer Inst 109, doi:10.1093/jnci/djx029.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1093/jnci/djx029&link_type=DOI) 

13. Barbera L, Sutradhar R, Howell D, et al. (2015) Does routine symptom screening with ESAS decrease ED visits in breast cancer patients undergoing adjuvant chemotherapy? Support Care Cancer 23:3025–32.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1007/s00520-015-2671-3&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=25711657&link_type=MED&atom=%2Fcmajo%2F7%2F1%2FE101.atom) 

14. Schepers SA, Sint Nicolaas SM, Haverman L, et al. (2017) Real-world implementation of electronic patient-reported outcomes in outpatient pediatric cancer care. Psychooncology 26:951–9.
    
    

15. Basch E, Barbera L, Kerrigan CL, et al. (2018) Implementation of patient-reported outcomes in routine medical care. Am Soc Clin Oncol Educ Book 38:122–34.
    
    

16. Patient Reported Outcomes Measures (PROMs) (National Health Service, Leeds (UK)) Available: [https://digital.nhs.uk/data-and-information/data-tools-and-services/data-services/patient-reported-outcome-measures-proms](https://digital.nhs.uk/data-and-information/data-tools-and-services/data-services/patient-reported-outcome-measures-proms). accessed 2018 Sept 15.
    
    

17. Nilsson E, Orwelius L, Kristenson M (2016) Patient-reported outcomes in the Swedish National Quality Registers. J Intern Med 279:141–53.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmajo%2F7%2F1%2FE101.atom) 

18. Bruera E, Kuehn N, Miller MJ, et al. (1991) The Edmonton Symptom Assessment System (ESAS): a simple method for the assessment of palliative care patients. J Palliat Care 7:6–9.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=1714502&link_type=MED&atom=%2Fcmajo%2F7%2F1%2FE101.atom) 

19. Chang VT, Hwang SS, Feuerman M (2000) Validation of the Edmonton Symptom Assessment Scale. Cancer 88:2164–71.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1002/(SICI)1097-0142(20000501)88:9<2164::AID-CNCR24>3.0.CO;2-5&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=10813730&link_type=MED&atom=%2Fcmajo%2F7%2F1%2FE101.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000086592200024&link_type=ISI) 

20. Dudgeon D, King S, Howell D, et al. (2012) Cancer Care Ontario’s experience with implementation of routine physical and psychological symptom distress screening. Psychooncology 21:357–64.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1002/pon.1918&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=21308858&link_type=MED&atom=%2Fcmajo%2F7%2F1%2FE101.atom) 

21. Gilbert JE, Howell D, King S, et al. (2012) Quality improvement in cancer symptom assessment and control: the Provincial Palliative Care Integration Project (PPCIP). J Pain Symptom Manage 43:663–78.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/j.jpainsymman.2011.04.028&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=22464352&link_type=MED&atom=%2Fcmajo%2F7%2F1%2FE101.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000303194400007&link_type=ISI) 

22. Pereira J, Green E, Molloy S, et al. (2014) Population-based standardized symptom screening: Cancer Care Ontario’s Edmonton Symptom Assessment System and performance status initiatives. J Oncol Pract 10:212–4.
    
    [FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6Mzoiam9wIjtzOjU6InJlc2lkIjtzOjg6IjEwLzMvMjEyIjtzOjQ6ImF0b20iO3M6MjA6Ii9jbWFqby83LzEvRTEwMS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

23. Iron K, Zagorski BM, Sykora K, et al. (2008) Living and dying in Ontario: an opportunity for improved health information — ICES investigative report (Institute for Clinical Evaluative Sciences, Toronto).
    
    

24. Robles SC, Marrett LD, Clarke EA, et al. (1988) An application of capture-recapture methods to the estimation of completeness of cancer registration. J Clin Epidemiol 41:495–501.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/0895-4356(88)90052-2&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=3367181&link_type=MED&atom=%2Fcmajo%2F7%2F1%2FE101.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=A1988N334600011&link_type=ISI) 

25. 1.  Jensen OM, 
    2.  Parkin DM, 
    3.  MacLennan R, 
    4.  et al.
    
    Clarke EA, Marrett LD, Kreiger N (1991) in Cancer registration principles and methods, Cancer registration in Ontario: a computer approach, eds Jensen OM, Parkin DM, MacLennan R, et al. (International Agency for Research on Cancer, Lyon (France)) IARC Scientific Publication No 95, pp 246–57.
    
    

26. Wilkins R (2001) PCCF + version 3G user’s guide (Geocodes/PCCF): automated geographic coding based on the Statistics Canada postal code conversions files including postal codes to June 2001 (Statistics Canada, Ottawa).
    
    

27. Deyo RA, Cherkin DC, Ciol MA (1992) Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases. J Clin Epidemiol 45:613–9.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/0895-4356(92)90133-8&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=1607900&link_type=MED&atom=%2Fcmajo%2F7%2F1%2FE101.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=A1992JA22600006&link_type=ISI) 

28. (2004) Data quality of the Discharge Abstract Database following the first-year implementation of ICD-10-CA/CCI — final report (Canadian Institute for Health Information, Ottawa).
    
    

29. (2003) The Johns Hopkins ACG® Case-Mix System, version 6.0 release notes (The Johns Hopkins University Bloomberg School of Public Health, Baltimore).
    
    

30. Sibley LM, Moineddin R, Agha MM, et al. (2010) Risk adjustment using administrative data-based and survey-derived methods for explaining physician utilization. Med Care 48:175–82.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1097/MLR.0b013e3181c16102&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=19927013&link_type=MED&atom=%2Fcmajo%2F7%2F1%2FE101.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000274081900013&link_type=ISI) 

31. Mittmann N, Liu N, Porter J, et al. (2014) Utilization and costs of home care for patients with colorectal cancer: a population-based study. CMAJ Open 2:E11–7.
    
    

32. Chiu M, Lebenbaum M, Lam K, et al. (2016) Describing the linkages of the immigration, refugees and citizenship Canada permanent resident data and vial statistics death registry to Ontario’s administrative health database. BMC Med Inform Decis Mak 16:135.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1186/s12911-016-0375-3&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=27769227&link_type=MED&atom=%2Fcmajo%2F7%2F1%2FE101.atom) 

33. Liang K-Y, Zeger SL (1986) Longitudinal data analysis using generalized linear models. Biometrika 73:13–22, doi:10.1093/biomet/73.1.13.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1093/biomet/73.1.13&link_type=DOI) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=A1986A734100002&link_type=ISI) 

34. Evans WK, Truscott R, Cameron E, et al. (2017) Lessons learned implementing a province-wide smoking cessation intitative in Ontario’s cancer centres. Curr Oncol 24:e185–90.
    
    

35. Pereira JL, Chasen MR, Molloy S, et al. (2016) Cancer care professionals’ attitudes toward systematic standardized symptom assessment and the Edmonton Symptom Assessment System after large-scale population-based implementation in Ontario, Canada. J Pain Symptom Manage 51:662–72.e8.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=26743641&link_type=MED&atom=%2Fcmajo%2F7%2F1%2FE101.atom) 

36. Korzeniowski M, Kalyvas M, Mahmud A, et al. (2016) Piloting prostate cancer patient-reported outcomesin clinical practice. Support Care Cancer 24:1983–90.
    
    

37. Brundage MD, Barbera L, McCallum F, et al. (Oct 31, 2018) A pilot evaluation of the expanded prostate cancer index composite for clinical practice (EPIC-CP) tool in Ontario. Qual Life Res doi:10.1007/s11136-018-2034-x, [Epub ahead of print].
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1007/s11136-018-2034-x&link_type=DOI) 

*   Copyright 2019, Joule Inc. or its licensors