Introduction
What is new?
- •
Significant deficits exist in the validation and reporting of algorithms used to identify patients within health administrative data.
- •
Misclassification error represents an important form of bias in research using health administrative databases.
- •
The modified Standards for Reporting of Diagnostic accuracy criteria reported here can be used to improve the quality of reporting in studies of health state (disease or conditions) identification validation.
- •
Future efforts should address criteria for conduct and reporting of research using health administrative data.
Health services and epidemiologic research are best conducted with population-level data. This helps ensure the appropriate estimation of incidence and prevalence rates, the minimization of referral bias, and the overall generalizability of the study conclusions to the population of interest. Because prospective clinical registries and retrospective chart review comprising a representative sample or all residents of a jurisdiction are impractical, health administrative data are an alternative for population-based chronic disease surveillance, outcomes research, and health services research. Health administrative data are defined as information passively collected, often by government and health care providers, for the purpose of managing the health care of patients [1], and are a subtype of automated health care data [2]. Examples include physician billing databases (such as those managed by government in single-payer health systems or by health maintenance organizations [HMOs]), and hospital discharge record databases. Accuracy of the diagnostic codes used to identify patients within these data depends on multiple factors including database quality, the specific condition being identified, and the validity of the codes in the patient group. A large gradient in data quality exists, with some databases being of higher quality than others [3]. Isolated diagnostic codes associated with physician billing records have been shown to be accurate to identify patients with some chronic diseases [4], [5] but not others [6], [7], [8], [9]. Since chronic diseases usually require multiple contacts with the health system to diagnose, a single-visit diagnostic code is often insufficient to accurately identify patients with the disease. The validity of codes is also dependent on the patient group being studied. For instance, the accuracy of diagnostic codes or combinations of codes (algorithms) varies across age groups because of variable use of the health system [6], [7], [10]. As such, validation of algorithms used to identify patients with different health states (including acute conditions, chronic diseases, and other health outcomes) is essential to avoid misclassification bias [11], which may threaten the internal validity and interpretation of study conclusions. For example, assessment of health services utilization in a cohort of patients with a chronic disease contaminated by large number of healthy residents falsely labeled as having a chronic disease would underestimate the burden of the disease on the health system or the quality and performance of the health system. Similarly, assessment of incidence of the disease in the cohort would overestimate risk to the population. Although the validation of administrative data coding has been identified as a priority in the health services research by an international consortium [3], the complete and accurate reporting of algorithm validation research is equally important to appropriate application. The growing availability of administrative data for research coupled with the expense, privacy concerns, and complex methodologies required to validate identification algorithms have resulted in algorithms being applied to these databases by researchers not involved in their initial validation. As such, minimum quality criteria for the conduct and reporting of algorithm validation studies would benefit scientists using these algorithms and consumers of the research on which these algorithms rely.
The purpose of this study was to appraise all studies that validated algorithms to identify patients with different health states within the administrative data with newly developed consensus criteria for the reporting of studies that validate health administrative data algorithms, based on the Standards for Reporting of Diagnostic accuracy (STARD) initiative [12]. In so doing, we aimed to identify strengths and weaknesses in the methods of such validation studies to improve the future reporting of research using health administrative data.