• Corpus ID: 150373798

Assuring the Machine Learning Lifecycle: Desiderata, Methods, and Challenges

  title={Assuring the Machine Learning Lifecycle: Desiderata, Methods, and Challenges},
  author={Rob Ashmore and Radu Calinescu and Colin Paterson},
Machine learning has evolved into an enabling technology for a wide range of highly successful applications. The potential for this success to continue and accelerate has placed machine learning (ML) at the top of research, economic and political agendas. Such unprecedented interest is fuelled by a vision of ML applicability extending to healthcare, transportation, defence and other domains of great societal importance. Achieving this vision requires the use of ML in safety-critical… 

Figures and Tables from this paper

MLife: a lite framework for machine learning lifecycle initialization

  • Cong YangWenfeng Wang John See
  • Computer Science
    2021 IEEE 8th International Conference on Data Science and Advanced Analytics (DSAA)
  • 2021
This work introduces a simple yet flexible framework, MLife, for fast ML lifecycle initialization, built on the fact that data flow in MLife is in a closed loop driven by bad cases, especially those which impact ML model performance the most but also provide the most value for further ML model development.

Guidance on the Assurance of Machine Learning in Autonomous Systems (AMLAS)

This document introduces a methodology for the Assurance of Machine Learning for use in Autonomous Systems (AMLAS), a process for systematically integrating safety assurance into the development of ML components and for generating the evidence base for explicitly justifying the acceptable safety of these components when integrated into autonomous system applications.

MLOps: A Taxonomy and a Methodology

This work reviews the existing scientific literature on ML Operations and proposes a taxonomy for clustering research papers on MLOps, and presents methodologies and operations aimed at defining a ML pipeline to simplify the release of ML applications in industry.

Machine Learning Operations: A Survey on MLOps Tool Support

There is a shortage in the availability of a fully functional MLOps platform on which processes can be automated by reducing human intervention and the importance of the Machine Learning Operations (MLOps) concept is discussed.

When Bioprocess Engineering Meets Machine Learning: A Survey from the Perspective of Automated Bioprocess Development

This review provides a comprehensive overview of ML-based automation in bioprocess development and identifies the missing links to enable the easy implementation of ML and Artificial Intelligence tools in valuable solutions for the bio-community.

Challenges in Deploying Machine Learning: A Survey of Case Studies

By mapping found challenges to the steps of the machine learning deployment workflow, it is shown that practitioners face issues at each stage of the deployment process.

Taxonomy of Machine Learning Safety: A Survey and Primer

The taxonomy of ML safety presents a safety-oriented categorization of ML techniques to provide guidance for improving dependability of the ML design and development and can serve as a safety checklist to aid designers in improving coverage and diversity of safety strategies employed in any given ML system.

Developing ML/DL Models: A Design Framework

Artificial Intelligence is becoming increasingly popular with organizations due to the success of Machine Learning and Deep Learning techniques. Using these techniques, data scientists learn from

Towards Addressing MLOps Pipeline Challenges: Practical Guidelines Based on a Multivocal Literature Review

Insight is provided into how certain MLOps challenges can be overcome by following guidelines (not tool specific) mentioned in the study in the area of research and industry.

Safety of Artificial Intelligence: A Collaborative Model

This paper sets out a three-layer model, going from top to bottom: system safety/functional safety; “AI/ML safety”; and safety-critical software engineering, and argues that this model should underpin future standards and guidelines for safety of this class of system which employ ML.



Data Lifecycle Challenges in Production Machine Learning

Challenges in data understanding, data validation and cleaning, and data preparation are explored - how different constraints are imposed on the solutions depending on where in the lifecycle of a model the problems are encountered and who encounters them are explored.

Accelerating the Machine Learning Lifecycle with MLflow

MLflow, an open source platform recently launched to streamline the machine learning lifecycle, covers three key challenges: experimentation, reproducibility, and model deployment, using generic APIs that work with any ML library, algorithm and programming language.

Using Machine Learning Safely in Automotive Software: An Assessment and Adaption of Software Process Requirements in ISO 26262

A detailed assessment and adaption of ISO 26262 for ML is done, specifically in the context of supervised learning, to address a conflict between the need to innovate and theneed to improve safety in automotive development.

DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems

DeepGauge is proposed, a set of multi-granularity testing criteria for DL systems, which aims at rendering a multi-faceted portrayal of the testbed and sheds light on the construction of more generic and robust DL systems.

Explainable artificial intelligence: A survey

Recent developments in XAI in supervised learning are summarized, a discussion on its connection with artificial general intelligence is started, and proposals for further research directions are given.

DeepCT: Tomographic Combinatorial Testing for Deep Learning Systems

  • L. MaFelix Juefei-Xu Jianjun Zhao
  • Computer Science
    2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER)
  • 2019
This paper proposes a set of combinatorial testing criteria specialized for DL systems, as well as a CT coverage guided test generation technique, and demonstrates that CT provides a promising avenue for testing DL systems.

Example and Feature importance-based Explanations for Black-box Machine Learning Models

This work presents a new explanation extraction method called LEAFAGE, for a prediction made by any black-box ML model, which consists of the visualization of similar examples from the training set and the importance of each feature, and aims to take the expectations of the user into account.

Machine Learning that Matters

This work presents six Impact Challenges to explicitly focus the field of machine learning's energy and attention, and discusses existing obstacles that must be addressed.

A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective

This survey performs a comprehensive study of data collection from a data management point of view, providing a research landscape of these operations, guidelines on which technique to use when, and identify interesting research challenges.

Machine learning - a probabilistic perspective

  • K. Murphy
  • Computer Science
    Adaptive computation and machine learning series
  • 2012
This textbook offers a comprehensive and self-contained introduction to the field of machine learning, based on a unified, probabilistic approach, and is suitable for upper-level undergraduates with an introductory-level college math background and beginning graduate students.