Background: High-quality evidence from randomised trials is a general requirement for the introduction of population-based screening programmes. However, for some types of screening no relevant data from randomised trials exist, while the intervention is nonetheless offered in nationwide programmes based on apparent beneficial effects seen in non-randomised studies. When performing a Cochrane review on screening for malignant melanoma, we ended up in a dilemma; should we include non-randomised studies in our review or should we ignore them?

Objectives: To find out how to deal with problems arising when high-quality evidence from randomised trials is lacking, while low-quality evidence from non-randomised studies has been used to justify a screening programme.

Method: Reflections based on experience from encountering this dilemma in a Cochrane review on screening for malignant melanoma.

Results: We had preplanned not to include non-randomised studies in our review. However, while working on the review we encountered a number of non-randomised studies frequently used to justify this type of screening. In the discussion part of the review, we chose post hoc to evaluate those non-randomised studies that had influenced national policy, using the ROBINS-I tool (Risk Of Bias In Non randomised Studies of Interventions).

Conclusion: Including non-randomised studies leads to a massive extra workload compared to restricting inclusion to randomised trials only, which will inevitably increase time to publication. Additionally, including data from non-randomised studies in a Cochrane review on population screening might legitimise the use of such interventions based on low-quality evidence for other screening programmes. However, if we do not assess the merits of studies that have been pivotal for far-reaching public health decisions, it would reduce the relevance of the corresponding Cochrane reviews. A compromise might be to evaluate the most relevant non-randomised studies in the discussion section of the review. However, to evaluate some non-randomised trials without performing a systematic search might introduce selection bias.