Background: Quality assessment of included studies is a crucial step when preparing systematic reviews. Although it is possible for reviewers to simply assess what they consider to be key components of risk of bias (ROB), this may result in important sources of bias being omitted, inappropriate items included or too much emphasis being given to particular items guided by reviewers’ subjective opinions. In contrast, a structured tool provides a convenient, standardised way to assess ROB providing consistency across reviews.

Objectives: To develop a framework for developing quality-assessment (QA) tools.

Methods: Based on our experiences of developing a variety of QA tools for studies of differing designs over the last 14 years, we have developed a suggested framework for developing QA tools.

Results: The framework consists of a three stages - (1) initial steps; (2) tool development; and, (3) dissemination. Each stage includes defined steps, which we consider important to follow when developing a tool. However, there is some flexibility on how these steps may be approached. In developing this framework we have drawn on our extensive experience of developing a number of QA tools including QUADAS-2 for diagnostic accuracy studies; ROBIS for systematic reviews; PROBAST for prediction modelling studies; ROBINS-I for non-randomised studies of interventions; and, the new version of the Cochrane risk-of-bias tool for randomised trials (RoB 2.0). Despite having used different approaches to the development of each of these tools, we found that all approaches shared common features and processes. This led to the development of this framework.

Conclusions: We recommend that anyone who would like to develop a new QA tool follow the stages outlined in this paper. We hope that our proposed framework will increase the number of tools developed using robust methods.