Background: The health effects and safety of non-nutritive sweeteners were investigated through various studies, with different study designs. ROBINS-I is proposed as a tool for quality assessment of non-randomised studies (non-RCTs).

Objectives: To assess the quality of non-RCTs with ROBINS-I, as part of a systematic review about health effects of non-nutritive sweeteners.

Methods: We used standard systematic review methodology as proposed by the Cochrane Handbook for the systematic review. Quality assessment of non-RCTs was conducted using ROBINS-I following the published guidance. A common understanding on how to assess each ROBINS-I domain was sought for in repeated discussions prior to assessment. Pairs of researchers independently assessed risk of bias in all domains and noted their results and comments in an Excel sheet. A harmonised understanding of the domains and the judgement was supported by discussing disagreements after initial assessment.

Results: We identified and included 60 studies in the systematic review. Of these, 32 were non-RCTs, including 9 non-RCTS or quasi-RCTs, 6 cohort studies, 16 case-control studies, and 1 cross-sectional study. Most non-RCTs had a serious risk of bias according to ROBINS-I assessment. A common understanding of the domains and judgement procedure during assessment had to be established by using the ROBINS-I guidance and during various group discussions.

Conclusions: The initial understanding of and the judgement with the ROBINS-I assessment tool varied slightly across researchers, but a common understanding could be established by using the guidance and group discussions. Comments and notes about common methodological procedures in nutritional research supported the judgement of the study quality. Guidance for the quality assessment of different study designs with ROBINS-I would be helpful in our perspective.