Potentially biasing limitations of included studies
| Investigator | Confounders, including cointerventions (ecological fallacy, confounding) | Bias in comparison groups (selection bias) | Data collection issues and missing data (selection bias, nonresponse bias, information biases including recall bias and reporting bias) | Lack of detail regarding intervention or implementation (bias due to deviation from or variation in interventions) | Outcome assessment methods or measures (measurement bias) |
|---|---|---|---|---|---|
| Averhoff et al. (23) | No measurement or adjustment for important potential confounders (e.g., home learning rates, noncompliance) | – | Self-reported vaccination data from single school district; response rate unknown | Exemption process, consequences for noncompliance and other implementation factors not specified | No external verification of vaccination status |
| Bugenske et al. (24) |
| May have been unobserved differences in individuals between states with and without mandates | Analysis limited to landline telephones and responses accompanied by provider-verified records; may not be representative | Policies were grouped together, not allowing for analysis of subtle differences in implementation or context | Follow-up time for policies limited; up-to-date vaccination status defined as 1 dose |
| Carpenter et al. (25) | Ecological study |
| Used 2008 data as proxy for premandate 2004/05 vaccination status; no middle school enrolment data; no premandate data | Multiple state policies grouped together; no accounting for differences | – |
| Cuff et al. (26) | No measurement or adjustment for important potential confounders | – | Single-centre study; low response rate; participants included only parents seeking care for well-child care visits; may not be representative | – | Only 1 yr of baseline (premandate) data |
| D’Ancona et al. (20) |
| – | – | – | Lack of reliable denominator; no testing for statistical significance of changes; only 1 yr of postmandate data |
| Jackson et al. (39) |
| – | Convenience sample of schoolchildren (not random) from a random sample of school districts | – | No external verification of vaccination status (parent report); no testing for statistical significance of changes; only 1 yr of pre- and postmandate data |
| Jacobs et al. (27) | Ecological study | May have been unobserved differences in individuals between states with and without mandates |
| Policies grouped together, not allowing for analysis of subtle differences in implementation or context | – |
| Karikari et al. (28) |
| – | Included only school-enrolled children in vaccination registry; 2 different data sources for outcome had different results; unclear why differences existed; lack of detail on CDC survey | No information on implementation or context | Unknown to what extent findings can be extrapolated to larger population |
| Kharbanda et al. (32) | No measurement or adjustment for potential confounders, although did look at spillover effect on nonmandate vaccination | – |
| No information on implementation or context | – |
| Morita et al. (29) | Ecological study | – | Losses to follow-up (e.g., students leaving school) excluded from analysis | Likely inconsistent enforcement of policy, not captured by study data collection methods | Only 2 yr of postmandate data |
| Moss et al. (30) |
| May have been unobserved differences in setting between states with and without mandates | – | Likely inconsistent enforcement of policy, not captured by study data collection methods | Unspecified/unadjusted for state differences in age/grade of mandate |
| Olshen et al. (31) | Ecological study | May have been differences in population with study insurer and population as a whole (representativeness and generalizability) | – | Policies grouped together, not allowing for subtle differences in implementation or context | Full model information not provided |
| Omer et al. (22) | Other known changes (e.g., in vaccination schedule, exemption forms) before policy change appear to have affected trends | – | Home learners may not have been included | – | – |
| Perkins et al. (33) | Ecological study | May have been unobserved differences in setting between states with and without mandates | Included only respondents with adequate provider-verified vaccination history | Policies grouped together, not allowing for subtle differences in implementation or context | Only 1 yr of baseline (premandate) data |
| Pierre-Victor et al. (34) | Ecological study | May have been unobserved differences in setting between states with and without mandates | Landline-only sample; analysis included only those who responded about HPV | – | – |
| Potter et al. (35) | Ecological study | – | Home learners may not have been included | – | Only 1 yr of baseline (premandate) and follow-up (postmandate) data |
| Simpson et al. (36) |
| – | Comparison with census data indicates registry may have underestimated coverage | – | – |
| Thompson et al. (21) | Ecological study | – | Data not available from nongovernmental schools; only schools with kindergarten enrolment included | Not possible to know reason for missing documentation, so unclear whether this represents bias in coverage outcome; some schools may have been more compliant than others | Limited pre- and postmandate data |
| Thompson et al. (37) |
| “All other states” comparator includes states both with and without mandates | Parent report only (no provider verification) | Implementation details not specified other than difficult to opt out | Only 1 yr of postmandate data |
| Wilson et al. (38) | Many cointerventions described; no measurement or control for potential confounders | Small school-based population may not be representative; combination of random and purposive sampling; 1 school excluded owing to improper documentation; nonenrolled students excluded (potential selection bias; enrolment in rural areas below target | – | Implementation details not specified | Small sample, insufficient statistical power |
Note: ACIP = Advisory Committee on Immunization Practices, CDC = Centers for Diseases Control and Prevention, HPV = human papillomavirus.