Background: The Risk of Bias In Non-Randomised Studies of Interventions (ROBINS-I) tool, and its associated guidance, has a clinical orientation, and does not yet address issues in specific NRS designs such as controlled before- and-after (CBA). Moreover, previous versions of the tool raised issues of applicability. There is a need to test the applicability of ROBINS-I for commonly used study designs, like CBA, and beyond the clinical realm, such as public health, where NRS often represent the best-available evidence.
Objective: To establish applicability of ROBINS-I for non-clinical interventions using a CBA design.
Methods: Five researchers, all experienced in critical appraisal of non-randomised studies, used ROBINS-I to assess risk of bias in 5 studies which had evaluated the health impacts of housing improvement; 4 using a CBA, and 1 a before-and-after design. ROBINS-I assessments for each study were entered into a database and checked for consensus across the group. Group discussions were used to identify reasons for lack of consensus for specific questions and bias domains.

Results: ROBINS-I helped to systematically articulate sources of bias in NRS, however, the lack of consensus in assessments across each of the 7 domains questioned its reliability and applicability to studies of natural experiments. The 2 domains with least consensus were: Selection (Domain 2); and Performance (Domain 4). These issues arose due to a lack of clarity for the unit of allocation of the intervention and analysis, as well as uncertainty of the time-point of ascertainment of intervention status which may conceal selection bias. This raised more fundamental difficulties when applying concepts which underpin ROBINS-I. Specifically, the definition of a pragmatic or explanatory trial, and their related Effects of Interest could not be applied to four studies.

Conclusion: Difficulties in applying ROBINS-I may be due to poor design and reporting of controlled before-and-after studies; this may improve in future. In the meantime, improved guidance on applying the tool is needed to allow existing evidence from natural experiments to be assessed appropriately.