Overview of software for screening large amounts of textual data implementing machine learning

The repository aims to create an overview and comparison of software used for systematically screening large amounts of textual data using machine learning.

Overview

The table below provides a quick overview of the software. The following properties are evaluated:

• Is there a website?
• Is the software open-source (provide a 🔗 to the source code)?
• Is the software peer-reviewed in a scientific article?
• Is documentation or a manual available (provide a 🔗)?
• Is the full version of the software free of charge?

Software	Website	Open-Source	Published	Documentation	Free
Abstrackr	🔗	❌		❌	✅
ASReview	🔗	✅🔗		✅🔗	✅
Colandr	🔗	❌		✅🔗	✅
DistillerSR	🔗	❌		✅🔗	❌
EPPI-Reviewer	🔗	❌	❌	✅🔗	❌
FASTREAD	❌	✅🔗		✅🔗	✅
Rayyan	🔗	❌		✅🔗	❌
RobotAnalyst	🔗	❌		❌	❔1
SWIFT-Active Screener	🔗	❌		✅🔗	❌

✅ Yes/Implemented; ❌ No/Not implemented; ❔ Unknown (requires an issue).

¹ See issue Rensvandeschoot#29

Installation

The table below provides an overview of options for how to install the software.

• Can the software be installed locally so that data and labeling decisions are only stored on the user's device (yes/no)?
• Is the software installable on a server (yes/no)?
• Is the software available as online service (software as a service - SAAS; yes/no; provide a link to the registration page)?

Software	Local	Server	Online Service
Abstrackr	❌	❌	✅🔗
ASReview	✅	✅	❌
Colandr	❌	❌	✅🔗
DistillerSR
EPPI-Reviewer	❌	❌	✅🔗
FASTREAD	✅	✅	❌
Rayyan	❌	❌	✅🔗
RobotAnalyst	❌	❌	✅🔗1
SWIFT-Active Screener	❌	❌	❌🔗

✅ Yes; ❌ No; ❔ Unknown (requires an issue).

¹ To use RobotAnalyst, you need to request an account via email.

Data

The table below provides an overview of input/output data.

• Which data formats can be imported?
• Can partly labeled data be imported (yes/no; if yes, as S(ingle) or M(ultiple) files)?
• Which data formats can be exported?
• Does the export file contain the labeling decisions?
• Can the export file be re-imported into the same software, retaining the labeling decisions (Re-Import-1: yes/no)?
• Can the export file be re-imported into reference manager software retaining, the labeling decision (Re-Import-2: yes/no)?

Software	Input data format	Partly labeled	Output data format	Labeling decisions	Re-Import-1	Re-Import-2
Abstrackr	RIS, TAB, TXT1	❌	CSV, XML, RIS	✅	❌	✅
ASReview	RIS, TSV, CSV, XLSX, TAB, +2	✅(S)+2	RIS, TSV, CSV, XLSX, TAB	✅	✅	✅
Colandr	RIS, BIB, TXT	✅(M)	CSV	✅	❌	❌
EPPI-Reviewer	RIS, TXT, +3	✅(M)	RIS, XLSX	❔4	❔4	❔4
FASTREAD	CSV	✅(S)	CSV	✅	✅	❌
Rayyan	RIS, ENW, BIB, CSV, XML, CIW, NBIB	✅(M)	RIS, BIB, ENW, CSV	✅	❌	✅
RobotAnalyst	RIS, NBIB	✅❔5	❔5	✅	❔5	❌
SWIFT-Active Screener	TXT, RIS, XML, BibTex	✅(M)	CSV, RIS	✅	❔6	✅

✅ Yes/Implemented; ❌ No/Not implemented; ⚡ Only for some extensions (add a footnote for more explanation); ❔ Unknown (requires an issue).

¹ List of PubMed IDs

² ASReview provides several open-source tools to convert file formats (e.g., CSV->RIS or RIS->XLSX), combine datasets (labeled, partly labeled, or unlabeled), and deduplicate records based on title/abstract/DOI.

³ EPPI-Reviewer provides a closed-source online file converter to convert several file formats to RIS.

⁴ See issue Rensvandeschoot#21

⁵ See issue Rensvandeschoot#29

⁶ See issue Rensvandeschoot#40

Machine Learning Properties

The tables below provide an overview of the machine learning properties.

Active Learning

Training Data

• Can training data (prior knowledge) be selected by the user to train the first iteration of the model (yes/no)?
• What is the minimal training data size (provide a number for Relevant and Irrelevant records)?

Software	Tr.Data by user	Minimum Tr.data
Abstrackr	❌	❔1
ASReview	✅	≥1R+≥1I
Colandr	✅	10
EPPI-Reviewer	✅	≥5R
FASTREAD	✅	≥1R
Rayyan	✅	≥50 with ≥5R
RobotAnalyst	✅	≥1R
SWIFT-Active Screener	✅2	≥1R3

✅ Yes/Implemented; ❌ No/Not implemented; ⚡ With some effort (add a footnote for more explanation); ❔ Unknown (requires an issue).

¹ See issue Rensvandeschoot#34

² Only relevant records can be provided as training data prior to screening.

³ If no relevant records are uploaded prior to screening, training will be initiated after screening ≥30 records with atleast ≥1R and ≥1I.

Model Selection

• Can the user select the active learning model (yes/no)?
• Can a user upload their own model (yes/no)?
• Can the feature extraction results be stored (yes/no)?
• Does (re-)training proceed Automatically or is it triggered Manually?
• Can the user continue labeling during training (yes/no)?
• Can the user select batch size (yes/no; provide the default)?
• Is it possible to switch to a different model during screening (yes/no)?

Software	Select model	User model	Store Feat.matrix	Training	Continue	Batch size	Switch
Abstrackr	❌	❌	❌	A	✅	❌	❌
ASReview	✅	✅	✅	A	✅	❌ (1)	⚡1
Colandr	❌	❌	❌	A	✅	❌ (10)	❌
EPPI-Reviewer	❌	❌	❌	M	✅	❌	❌
FASTREAD	❌	❌	❌	M	❌	❌	❌
Rayyan	❌	❌	❌	M	✅	❌	❌
RobotAnalyst	❌	❌	❌	M	❔2	❌	❌
SWIFT-Active Screener	❌	❌	❌	A	❔3	❌ (30)	❌

✅ Yes/Implemented; ❌ No/Not implemented; ⚡ With some effort (add a footnote with more explanation);

¹ Switching to a different model in ASReview is available by exporting the data of the first model and importing the data back into ASReview. The software will recognize all previous labeling decisions, and a new model can be trained.

² See issue Rensvandeschoot#29

³ See issue Rensvandeschoot#40

Overview of Available Models

• Which feature extraction methods are available? BOW = bag of words; Doc2Vec = document to vector; sBERT = sentence bidirectional encoder representations from transformers; TF–IDF = term frequency–inverse document frequency; Word2Vec = words to vector; ML = Multi-language;

• Which classifiers are available? CNN = convolutional neural network; DNN = dense neural network; LDA = latent Dirichlet allocation; LL = log linear; LR= logistic regression; LSTM = long short-term memory; NB = naive Bayes; RF =random forests; SGD = stochastic gradient descent; SVM = support vector machine;

• Which balancing strategies are available? S / Simple = no balancing balance strategy; D / Double = Double balance strategy; T / Triple = Triple balance strategy; U / Under = Undersampling balance strategy; A / Aggressive = Aggressive undersampling balance strategy (after classifier is stable); W / Weighting = Weighting for data balancing (before and after classifier is stable); M / Mixing = Mixing: weighting is applied before the classifier is stable and aggressive undersampling is applied after the classifier is stable;

• Which query strategies are available? R / Random = Records are selected randomly; C / Certain = Certainty based; U / Uncertain = Uncertainty based; M / Mixed = A combination of query strategies, for example 90% Certainty based and 10% Random; Cl / Clustering = Clustering query strategy;

Software	Feature Extr.	Classifiers	Balancing	Query Stra.
Abstrackr	TF-IDF ❔1	SVM	❔1	R, C, U
ASReview	TF–IDF, Doc2Vec, sBert, TF-IDF, ML	CNN, DNN, LR, LSTM, NB, RF, SVM	S, D, U, T	R, C, U, M, CL
Colandr	Word2Vec ❔2	SGD ❔ 2	❔2	C
EPPI-Reviewer	TF-IDF	SVM	❔3	R, C, Cl
FASTREAD	TF-IDF	SVM	S, A, W, M	C, U
Rayyan	❔4	SVM	❔4	C, U
RobotAnalyst	TF-IDF + BOW + LDA2vec	SVM	❔5	R, C, U, Cl
SWIFT-Active Screener	TF-IDF	LL	S:grey_question:6	C

✅ Yes/Implemented; ❌ No/Not implemented; ❔ Unknown (requires an issue).

¹ See issue Rensvandeschoot#34

² See issue Rensvandeschoot#16

³ See issue Rensvandeschoot#21

⁴ See issue Rensvandeschoot#19

⁵ See issues Rensvandeschoot#29

⁶ See issues Rensvandeschoot#40

Supervised Learning

Software	Feature Extr.	Classifiers	Balancing	Query Stra.
EPPI-Reviewer1	TF-IDF	SVM:grey_question:2	❔2	R, C, Cl

¹ EPPI-Reviewer offers the option to choose from, or use custom, pre-trained models to find a specific type of literature, e.g., for RCTs.

² See issue Rensvandeschoot#21

Unsupervised Learning

Software	Q1

Software

This section briefly describes the software in alphabetical order.

Abstrackr

Abstrackr is a collaborative (i.e., multiple reviewers can simultaneously screen citations for a review), web-based annotation tool for the citation screening task.

ASReview

ASReview, developed at Utrecht University, helps scholars and practitioners to get an overview of the most relevant records for their work as efficiently as possible while being transparent in the process. It allows multiple machine learning models, and ships with exploration and simulation modes, which are especially useful for comparing and designing algorithms. Furthermore, it is intended to be easily extensible, allowing third parties to add modules that enhance the pipeline with new models, data, and other extensions.

Colandr

Colandr is a free, web-based, open-access tool for conducting evidence synthesis projects.

DistillerSR

DistillerSR automates the management of literature collection, screening, and assessment using AI and intelligent workflows. From a systematic literature review to a rapid review to a living review, DistillerSR makes any project simpler to manage and configure to produce transparent, audit-ready, and compliant results.

EPPI-Reviewer

EPPI-Reviewer is a web-based software program for managing and analysing data in literature reviews. It has been developed for all types of systematic review (meta-analysis, framework synthesis, thematic synthesis etc) but also has features that would be useful in any literature review. It manages references, stores PDF files and facilitates qualitative and quantitative analyses such as meta-analysis and thematic synthesis. It also contains some new ‘text mining’ technology which is promising to make systematic reviewing more efficient.

FASTREAD

FASTREAD (FAST2) is a tool to support primary study selection in systematic literature review.

Rayyan

Rayyan is a free web and mobile app, that helps expedite the initial screening of abstracts and titles using a process of semi-automation while incorporating a high level of usability.

RobotAnalyst

RobotAnalyst was developed as part of the Supporting Evidence-based Public Health Interventions using Text Mining project to support the literature screening phase of systematic reviews.

SWIFT-Active Screener

SWIFT-Active Screener (SWIFT is an acronym for “Sciome Workbench for Interactive computer-Facilitated Text-mining”) is a freely available interactive workbench which provides numerous tools to assist with problem formulation and literature prioritization.

Contributing

Do you know other software that meets the inclusion criteria? Please make a Pull Request and add it to the overview. When there is missing, wrong, or incomplete information, please start an issue.

Licence

This project is CC-BY 4.0 licensed.

Contact

For any suggestions, questions, or remarks, please file an issue in the issue tracker.

This comparison is maintained by Rens van de Schoot. I aim to make a fair comparison and not to be prejudiced. If there is any concern about the comparison, please file an issue in the issue tracker such that it can be openly discussed.