Estimating effects of health policy interventions using interrupted time-series analyses: a simulation study

TY - JOUR

T1 - Estimating effects of health policy interventions using interrupted time-series analyses: a simulation study

AU - Jiang, Huan

AU - Feng, Xinyang

AU - Lange, Shannon

AU - Tran, Alexander

AU - Manthey, Jakob

AU - Rehm, Jürgen

N1 - © 2022. The Author(s).

PY - 2022/8/31

Y1 - 2022/8/31

N2 - BACKGROUND: A classic methodology used in evaluating the impact of health policy interventions is interrupted time-series (ITS) analysis, applying a quasi-experimental design that uses both pre- and post-policy data without randomization. In this paper, we took a simulation-based approach to estimating intervention effects under different assumptions.METHODS: Each of the simulated mortality rates contained a linear time trend, seasonality, autoregressive, and moving-average terms. The simulations of the policy effects involved three scenarios: 1) immediate-level change only, 2) immediate-level and slope change, and 3) lagged-level and slope change. The estimated effects and biases of these effects were examined via three matched generalized additive mixed models, each of which used two different approaches: 1) effects based on estimated coefficients (estimated approach), and 2) effects based on predictions from models (predicted approach). The robustness of these two approaches was further investigated assuming misspecification of the models.RESULTS: When one simulated dataset was analyzed with the matched model, the two analytical approaches produced similar estimates. However, when the models were misspecified, the number of deaths prevented, estimated using the predicted vs. estimated approaches, were very different, with the predicted approach yielding estimates closer to the real effect. The discrepancy was larger when the policy was applied early in the time-series.CONCLUSION: Even when the sample size appears to be large enough, one should still be cautious when conducting ITS analyses, since the power also depends on when in the series the intervention occurs. In addition, the intervention lagged effect needs to be fully considered at the study design stage (i.e., when developing the models).

AB - BACKGROUND: A classic methodology used in evaluating the impact of health policy interventions is interrupted time-series (ITS) analysis, applying a quasi-experimental design that uses both pre- and post-policy data without randomization. In this paper, we took a simulation-based approach to estimating intervention effects under different assumptions.METHODS: Each of the simulated mortality rates contained a linear time trend, seasonality, autoregressive, and moving-average terms. The simulations of the policy effects involved three scenarios: 1) immediate-level change only, 2) immediate-level and slope change, and 3) lagged-level and slope change. The estimated effects and biases of these effects were examined via three matched generalized additive mixed models, each of which used two different approaches: 1) effects based on estimated coefficients (estimated approach), and 2) effects based on predictions from models (predicted approach). The robustness of these two approaches was further investigated assuming misspecification of the models.RESULTS: When one simulated dataset was analyzed with the matched model, the two analytical approaches produced similar estimates. However, when the models were misspecified, the number of deaths prevented, estimated using the predicted vs. estimated approaches, were very different, with the predicted approach yielding estimates closer to the real effect. The discrepancy was larger when the policy was applied early in the time-series.CONCLUSION: Even when the sample size appears to be large enough, one should still be cautious when conducting ITS analyses, since the power also depends on when in the series the intervention occurs. In addition, the intervention lagged effect needs to be fully considered at the study design stage (i.e., when developing the models).

KW - Computer Simulation

KW - Health Policy

KW - Humans

KW - Interrupted Time Series Analysis

KW - Research Design

KW - Sample Size

U2 - 10.1186/s12874-022-01716-4

DO - 10.1186/s12874-022-01716-4

M3 - SCORING: Journal article

C2 - 36045338

VL - 22

JO - BMC MED RES METHODOL

JF - BMC MED RES METHODOL

SN - 1471-2288

IS - 1

M1 - 235

ER -