Evaluating policies and programmes: An exploration of how the choice of non-randomised study design influences results

ID: 

2044

Session: 

Poster session 2 Thursday: Evidence synthesis - methods / improving conduct and reporting

Date: 

Thursday 14 September 2017 - 12:30 to 14:00

Location: 

Exhibition Halls 1 & 2

All authors in correct order:

Burns J1, Polus S1, Boogaard H2, Rehfuess E1
1 Institute for Medical Informatics, Biometry and Epidemiology, University of Munich, Germany
2 Health Effects Institute, USA
Presenting author and contact person

Presenting author:

Jacob Burns

Contact person:

Jacob Burns
Abstract text
Background:For individuals and populations to benefit from policies and programmes implemented in health, education, social welfare, environment or other sectors, it is critical that their effectiveness be rigorously evaluated. As non-randomised studies (NRS) are increasingly applied in evaluations and included in systematic reviews, researchers, systematic reviewers and decision makers need to understand their strengths and limitations. However, the most common NRS included in systematic reviews of effectiveness, controlled before-after (CBA) and interrupted time series (ITS) studies, tend to be inconsistently defined, employed and interpreted.
Objectives: In order to explore how the design and analysis can influence results of primary studies, and in turn potentially results and conclusions of systematic reviews, we conducted a series of re-analyses of a study included in a Cochrane review of interventions to reduce ambient air pollution.

Methods: We obtained the original data of the study 'Mortality Effects of a Copper Smelter Strike and Reduced Ambient Sulfate Particulate Matter Air Pollution' by Pope et al. 2007, which assessed the mortality effects associated with a copper smelter strike in the United States. Based on pre-specified study design definitions and statistical methods, we re-analysed the data as a controlled ITS, an uncontrolled ITS, a CBA and an uncontrolled before-after study. We also assessed how the choice of time period and control group affected results.

Results: Various study design aspects, i.e. inclusion of a control group, choice of control group, adjustment for underlying time trends and variations in the pre- and post-intervention periods, led to differential intervention effects, affecting the magnitude, significance, and in some cases direction.

Conclusions: The primary study design can substantially influence the estimated intervention effect. As NRS designs are applied to evaluate policies and programmes across different sectors, it is key that researchers utilise the most-reliable NRS designs, and that both researchers and reviewers carefully assess the risk of bias associated with the design and other design aspects.