Corpus ID: 208857578

Merlin: Enabling Machine Learning-Ready HPC Ensembles

  title={Merlin: Enabling Machine Learning-Ready HPC Ensembles},
  author={J. Peterson and Rushil Anirudh and Kevin Athey and Benjamin Bay and P. Bremer and Vic Castillo and F. Natale and D. Fox and J. Gaffney and D. Hysom and S. Jacobs and B. Kailkhura and J. Koning and B. Kustowski and Steven Langer and P. Robinson and Jessica L. Semler and B. Spears and J. Thiagarajan and B. V. Essen and Jae-Seung Yeom},
  • J. Peterson, Rushil Anirudh, +18 authors Jae-Seung Yeom
  • Published 2019
  • Computer Science, Physics
  • ArXiv
  • With the growing complexity of computational and experimental facilities, many scientific researchers are turning to machine learning (ML) techniques to analyze large scale ensemble data. With complexities such as multi-component workflows, heterogeneous machine architectures, parallel file systems, and batch scheduling, care must be taken to facilitate this analysis in a high performance computing (HPC) environment. In this paper, we present Merlin, a workflow framework to enable large ML… CONTINUE READING
    1 Citations


    Parallelizing Training of Deep Generative Models on Massive Scientific Datasets
    • 6
    • PDF
    Flux: Overcoming Scheduling Challenges for Exascale Workflows
    • 8
    • Highly Influential
    • PDF
    Pegasus, a workflow management system for science automation
    • 473
    • PDF
    FireWorks: a dynamic workflow system designed for high‐throughput applications
    • 206
    • Highly Influential
    • PDF
    A HYDRA UQ Workflow for NIF Ignition Experiments
    • 12
    Computing Just What You Need: Online Data Analysis and Reduction at Extreme Scales
    • 44
    • PDF
    Swift: A language for distributed parallel scripting
    • 364
    • Highly Influential
    • PDF
    Searching for exotic particles in high-energy physics with deep learning.
    • 678
    • PDF