• Corpus ID: 238259637

MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluation

  title={MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluation},
  author={Alexandros Karargyris and Renato Umeton and Micah J. Sheller and Alejandro S{\'a}nchez Aristiz{\'a}bal and Johnu George and Srini Bala and Daniel J. Beutel and Victor Bittorf and Akshay Chaudhari and Alexander Chowdhury and Cody Coleman and Bala Desinghu and Gregory Frederick Diamos and Debo Dutta and Diane Feddema and Grigori Fursin and Junyi Guo and Xinyuan Huang and David Kanter and Satyananda Kashyap and Nicholas D. Lane and Indranil Mallick and Pietro Mascagni and Virendra Mehta and Vivek Natarajan and Nikolay Nikolov and Nicolas Padoy and Gennady Pekhimenko and Vijay Janapa Reddi and G. Anthony Reina and Pablo Ribalta and Jacob Rosenthal and Abhishek Singh and Jayaraman J. Thiagarajan and Anne Wuest and Maria Xenochristou and Daguang Xu and Poonam Yadav and Michael Rosenthal and Massimo Loda and Jason M. Johnson and Peter Mattson},
Medical AI has tremendous potential to advance healthcare by supporting the evidence-based practice of medicine, personalizing patient treatment, reducing costs, and improving provider and patient experience. We argue that unlocking this potential requires a systematic way to measure the performance of medical AI models on large-scale heterogeneous data. To meet this need, we are building MedPerf, an open framework for benchmarking machine learning in the medical domain. MedPerf will enable… 
1 Citations

Figures and Tables from this paper

Developing medical imaging AI for emerging infectious diseases

This review aims to examine the strengths and weaknesses of prior studies and provide recommendations for different stages of building useful AI models for medical imaging, among them: needfinding, dataset curation, model development and evaluation, and post-deployment considerations.



Interactive machine learning for health informatics: when do we need the human-in-the-loop?

Interactive machine learning (iML) is defined as “algorithms that can interact with agents and can optimize their learning behavior through these interactions, where the agents can also be human.”

Federated learning for predicting clinical outcomes in patients with COVID-19.

This study facilitated rapid data science collaboration without data exchange and generated a model that generalized across heterogeneous, unharmonized datasets for prediction of clinical outcomes in patients with COVID-19, setting the stage for the broader use of FL in healthcare.

End-to-end privacy preserving deep learning on multi-institutional medical imaging

PriMIA (Privacy-preserving Medical Image Analysis), a free, open-source software framework for differentially private, securely aggregated federated learning and encrypted inference on medical imaging data, is presented.

Pancreatic cancer risk predicted from disease trajectories using deep learning

These results raise the state-of-the-art level of performance of cancer risk prediction on real-world data sets and provide support for the design of prediction-surveillance programs based on risk assessment in a large population followed by affordable surveillance of a relatively small number of patients at highest risk.

OpenFL: An open-source framework for Federated Learning

The first use of the OpenFL framework to train consensus ML models in a consortium of international healthcare organizations, as well as how it facilitates the first computational competition on FL are described.

The Federated Tumor Segmentation (FeTS) Challenge

This manuscript describes the first challenge on Federated Learning, namely the Federated Tumor Segmentation (FeTS) challenge 2021, which uses clinically acquired, multi-institutional magnetic resonance imaging (MRI) scans from the BraTS 2020 challenge, as well as from various remote independent institutions included in the collaborative network of a real-world federation.

Artificial intelligence in oncology: Path to implementation

Concrete actions that major stakeholders can take to overcome barriers to AI implementation in oncology include training and educating the oncologists workforce in AI; standardizing data, model validation methods, and legal and safety regulations; and developing, studying, and deploying AI tools through multidisciplinary collaboration.

Artificial intelligence-enabled electrocardiograms for identification of patients with low ejection fraction: a pragmatic, randomized clinical trial.

The trial met the prespecified primary endpoint, demonstrating that the intervention increased the diagnosis of low EF in the overall cohort and use of an AI algorithm based on ECGs can enable the early diagnosis ofLow EF in patients in the setting of routine primary care.