Background: Text-mining technologies have great potential to reduce reference screening burden in systematic reviews. RobotAnalyst, a web-based tool for text mining and automatic classification, seems promising for assisting with screening in Cochrane reviews. However, it remains unclear whether and how barriers such as complex search topics, large reference-retrieval datasets, search strategies with high recall and low precision, poorly or recently indexed references, and limited experience of reviewers impact on RobotAnalyst performance as compared to manual screening.
Objectives: 1) To assess RobotAnalyst performance in identifying eligible references and reducing screening burden and time for reviewers. 2) To provide an account of reviewers’ experience with RobotAnalyst usability.
Methods: We extracted references (title and abstract) from 6 databases for a review on measures of older inpatient safety based on administrative health data. Semi-automatic screening supported by RobotAnalyst is being performed by a junior and senior researcher. Its performance will be compared to manual screening by 2 senior researchers using standard Cochrane methodology. Yield, burden and median decision time will be measured during screening. Perceived usability of RobotAnalyst application will also be evaluated.
Results: Of 4964 extracted references, about 680 (13.7%) should be eligible for full-text retrieval. First results suggest that automatic classification has helped screen most of eligible references in the first phase of the review while reducing decision time from around 100 to < 20 seconds in the second phase (Figure 1). Manual and semi-automatic screenings by senior researchers are ongoing and performance data will be completed. RobotAnalyst usability could be improved by enabling reference de-duplication and annotation. Accounting for author list, tables/figures, and pre-specified keywords could also increase screening performance.
Conclusions: Our evaluative study should help systematic reviewers decide on whether using a text-mining tool, such as RobotAnalyst, is worthwhile for complex literature searches in public-health or health-services research.
