Deploying an interactive machine learning system in an evidence-based practice center: abstrackr

@inproceedings{Wallace2012DeployingAI,
  title={Deploying an interactive machine learning system in an evidence-based practice center: abstrackr},
  author={Byron C. Wallace and Kevin Small and Carla E. Brodley and Joseph Lau and Thomas A. Trikalinos},
  booktitle={International Health Informatics Symposium},
  year={2012}
}
Medical researchers looking for evidence pertinent to a specific clinical question must navigate an increasingly voluminous corpus of published literature. [] Key Method More specifically, we have developed abstrackr, an online tool for the task of citation screening for systematic reviews. This tool provides an interface to our machine learning methods. The main aim of this work is to provide a case study in deploying cutting-edge machine learning methods that will actually be used by experts in a clinical…

Figures from this paper

Machine Learning Approaches to Retrieve High-Quality, Clinically Relevant Evidence From the Biomedical Literature: Systematic Review (Preprint)

A systematic review of studies that applied machine learning techniques to identify high- quality clinical articles in the biomedical literature shows that machine learning approaches perform well in retrieving high-quality clinical studies.

Machine learning in health informatics: making better use of domain experts

The immediate aim in this work is to reduce the workload involved in conducting systematic reviews, and it is demonstrated that the developed methods can reduce reviewer workload by more than half, without sacrificing the comprehensiveness of reviews.

Machine learning for screening prioritization in systematic reviews: comparative performance of Abstrackr and EPPI-Reviewer

Screening prioritization functionality is a promising modality offering efficiency gains without giving up human involvement in the screening process, as compared with machine learning tools for potential use in citation screening.

Machine Learning Approaches to Retrieve High-Quality, Clinically Relevant Evidence From the Biomedical Literature: Systematic Review

A systematic review of studies that applied machine learning techniques to identify high-quality clinical articles in the biomedical literature shows that machine learning approaches perform well in retrieving high- quality clinical studies.

Machine learning algorithms for systematic review: reducing workload in a preclinical review of animal studies and reducing human screening error

This work has confirmed the performance and application ofML algorithms for screening in systematic reviews of preclinical animal studies and highlighted the novel use of ML algorithms to identify human error.

An open source machine learning framework for efficient and transparent systematic reviews

An open source machine learning-aided pipeline applying active learning: ASReview is developed and it is demonstrated by means of simulation studies that active learning can yield far more efficient reviewing than manual reviewing while providing high quality.

Research Screener: a machine learning tool to semi-automate abstract screening for systematic reviews

Initial evidence is introduced on the utility of Research Screener, a semi-automated machine learning tool to facilitate abstract screening and suggests that analysts who scan 50% of the total pool of articles identified via a systematic search are highly likely to have identified 100% of eligible papers.

srBERT: automatic article classification model for systematic review using BERT

The research shows the possibility of automatic article classification using machine-learning approaches to support SR tasks and its broad applicability, however, because the performance of the model depends on the size and class ratio of the training dataset, it is important to secure a dataset of sufficient quality, which may pose challenges.

ASK-the-Expert: Active Learning Based Knowledge Discovery Using the Expert

ASK-the-Expert is an interactive tool that allows SMEs to review instances from a data set and provide labels within a single framework and is powered by an active learning algorithm for training a classifier in the backend.

Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the Abstrackr machine learning tool

Abstrackr’s reliability and the workload savings varied by screening task, and how this might affect the results and conclusions of SRs needs to be evaluated.
...

References

SHOWING 1-10 OF 15 REFERENCES

Modeling annotation time to reduce workload in comparative effectiveness reviews

ABSTRACKR, an open-source annotation tool for assisting reviewers with the tedious task of citation screening for CERs, is presented and a strategy that integrates labeled terms and timing data into the Active Learning (AL) framework is proposed, in which an algorithm selects citations for the reviewer to label.

A Prospective Evaluation of an Automated Classification System to Support Evidence-based Medicine and Systematic Review.

A prospective study of a support vector machine-based classifier for supporting the SR literature triage process demonstrates that these methods can achieve accurate results in near-real world conditions and are promising tools for deployment to groups conducting SRs.

Who Should Label What? Instance Allocation in Multiple Expert Active Learning

A novel algorithm is developed that exploits the meta-cognitive abilities of novice (cheap) experts in order to make the best use of the experienced (expensive) annotators in this multiple expert active learning (MEAL) scenario.

Semi-automated screening of biomedical citations for systematic reviews

A semi-automated citation screening algorithm for systematic reviews has the potential to substantially reduce the number of citations reviewers have to manually screen, without compromising the quality and comprehensiveness of the review.

Active Learning Literature Survey

This report provides a general introduction to active learning and a survey of the literature, including a discussion of the scenarios in which queries can be formulated, and an overview of the query strategy frameworks proposed in the literature to date.

A Unified Approach to Active Dual Supervision for Labeling Features and Examples

Empirical results confirm that appropriately querying for both example and feature labels significantly reduces overall human effort--beyond what is possible through traditional one-dimensional active learning.

Support Vector Machine Active Learning with Applications to Text Classification

Experimental results showing that employing the active learning method can significantly reduce the need for labeled training instances in both the standard inductive and transductive settings are presented.

How well does active learning actually work? Time-based evaluation of cost-reduction strategies for language documentation.

It is found that better example selection and label suggestions improve efficiency, but effectiveness depends strongly on annotator expertise, and the need for cost-sensitive learning methods that adapt to annotators is stressed.

Frontiers of biomedical text mining: current progress

The current state of the art in biomedical text mining or 'BioNLP' in general is reviewed, focusing primarily on papers published within the past year.

The Constrained Weight Space SVM: Learning with Ranked Features

This work presents a novel classification model for exploiting domain knowledge by exploiting binary labeled features in the Constrained Weight Space SVM, and shows that the CW-SVM outperforms both baseline supervised learning strategies and previously proposed methods for learning with labeled features.