Background:One of the most time-consuming aspects of producing health technology assessment (HTA) reports is the task of identifying relevant studies in a way that minimises the risk of bias. New and emergent technologies are needed to achieve more efficient working processes without the loss of high standards of quality. The SBU Enquiry Service responds to questions posed by decision makers in healthcare and social services in Sweden, and gives a guide to the scientific literature of the field. Identified studies are not subject to formal quality control or evidence grading. Text mining is a technology that can be used to rank abstracts according to the assessor’s perception of relevance. Previous evaluation of Rayyan® has shown that a high number of the abstracts selected for reading in full text were identified early in the screening process.

Objectives:Our objective is to continue to evaluate relevance ranking in the abstract-screening process.

Methods:Rayyan is used by the SBU Enquiry Service for screening of abstracts. After going through 25, 50, 75 and a 100 per cent of references identified in the literature search, the number of potentially relevant abstracts are logged.

Results:So far we have completed evaluations of ten questions to the Enquiry Service. After screening half of the search result at least 88 percent of the relevant abstracts were identified, and for five of the projects all relevant abstracts were identified. After screening three quarters of the search result at least 95 percent of relevant abstracts were identified, and in seven projects all relevant abstracts were identified.

Conclusions:A majority of relevant abstracts are identified early in the screening process when using Rayyan® relevance ranking. It is important to continue to evaluate Rayyan’s relevance ranking, to add data from different knowledge fields.