Experiences with Improving the Transparency of AI Models and Services

@article{Hind2020ExperiencesWI,
  title={Experiences with Improving the Transparency of AI Models and Services},
  author={M. Hind and Stephanie Houde and Jacquelyn Martino and A. Mojsilovic and David Piorkowski and John T. Richards and K. Varshney},
  journal={Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems},
  year={2020}
}
AI models and services are used in a growing number of high-stakes areas, resulting in a need for increased transparency. Consistent with this, several proposals for higher quality and more consistent documentation of AI data, models, and systems have emerged. Little is known, however, about the needs of those who would produce or consume these new forms of documentation. Through semi-structured developer interviews, and two document-creation exercises, we have assembled a clearer picture of… Expand
A Methodology for Creating AI FactSheets
TLDR
This work describes a methodology for creating the form of AI documentation called FactSheets and describes the issues to consider and the questions to explore with the relevant people in an organization who will be creating and consuming the AI facts in a FactSheet. Expand
Towards evaluating and eliciting high-quality documentation for intelligent systems
TLDR
A set of quality dimensions is proposed and evaluated to identify in what ways this type of documentation falls short and how such dimensions can be use to further enable users to provide documentation that is suitable to a given persona or use case. Expand
Know Your Model (KYM): Increasing Trust in AI and Machine Learning
TLDR
This paper analyzes each element of trustworthiness and provides a set of 20 guidelines that can be leveraged to ensure optimal AI functionality while taking into account the greater ethical, technical, and practical impacts to humanity. Expand
Z-Inspection®: A Process to Assess Trustworthy AI
TLDR
This article outlines a novel process based on applied ethics, namely, Z-Inspection®, to assess if an AI system is trustworthy, and is the first process to assess trustworthy AI in practice. Expand
Do's and Don'ts for Human and Digital Worker Integration
TLDR
This position paper argues for a broader view that incorporates the potential for multiple levels of autonomy and human involvement, as well as a wider range of metrics beyond productivity when integrating digital workers into a business process. Expand
Challenges and Future Directions for Accountable Machine Learning
In recent years, machine learning (ML) algorithms have been applied in many areas such as healthcare, finance and autonomous vehicles. At the same time, there is an increasing need for making MLExpand
Yes We Care! - Certification for Machine Learning Methods through the Care Label Framework
TLDR
A unified framework that certifies learning methods via care labels is proposed that considers both, the machine learning theory and a given implementation and test the implementation’s compliance with theoretical properties and bounds. Expand
Documenting Computer Vision Datasets: An Invitation to Reflexive Data Practices
TLDR
This paper identifies four key issues that hinder the documentation of image datasets and the effective retrieval of production contexts and proposes reflexivity, understood as a collective consideration of social and intellectual factors that lead to praxis, as a necessary precondition for documentation. Expand
Fifty Shades of Grey: In Praise of a Nuanced Approach Towards Trustworthy Design
TLDR
It is argued that certain features which are commonly seen to benefit trust appear applicable to promoting trust in and through Virtual Labs, but whether they promote trust is a function of how systematically designers consider various (potentially conflicting) stakeholder trust needs. Expand
How AI Developers Overcome Communication Challenges in a Multidisciplinary Team
TLDR
Using the analytic lens of shared mental models, this paper reports on the types of communication gaps that AI developers face, how AI developers communicate across disciplinary and organizational boundaries, and how they simultaneously manage issues regarding trust and expectations. Expand
...
1
2
...

References

SHOWING 1-10 OF 59 REFERENCES
On-demand Developer Documentation
TLDR
The major challenges in realizing a paradigm shift in supporting the information needs of developers are discussed, existing research that can be leveraged to this end are highlighted, and opportunities for increased convergence in research on software documentation are promoted. Expand
The relevance of software documentation, tools and technologies: a survey
This paper highlights the results of a survey of software professionals. One of the goals of this survey was to uncover the perceived relevance (or lack thereof) of software documentation, and theExpand
A field study of API learning obstacles
TLDR
This study conducted a mixed approach, multi-phased study of the obstacles faced by Microsoft developers learning a wide variety of new APIs, with a special focus on obstacles related to API documentation. Expand
Application Programming Interface Documentation: What Do Software Developers Want?
TLDR
The results show that developers initially try to form a global understanding regarding the overall purpose and main features of an API, but then adopt either a concepts-oriented or a code-oriented learning strategy that API documentation both needs to address. Expand
Increasing Trust in AI Services through Supplier's Declarations of Conformity
TLDR
This paper envisiones an SDoC for AI services to contain purpose, performance, safety, security, and provenance information to be completed and voluntarily released by AI service providers for examination by consumers. Expand
How API Documentation Fails
TLDR
Researchers investigated how 10 common documentation problems manifested themselves in practice and found the three severest problems were ambiguity, incompleteness, and incorrectness of content. Expand
On the Comprehension of Program Comprehension
TLDR
It is found that developers follow pragmatic comprehension strategies depending on context and call for reconsidering the research agendas towards context-aware tool support. Expand
A study of the effectiveness of usage examples in REST API documentation
TLDR
By following the REST API documentation suggestions from this paper, REST API developers can reduce the errors, improve success rate and satisfaction of API client developers. Expand
Live API documentation
TLDR
Baker can be used to enhance traditional API documentation with up-to-date source code examples and incorporate links to the API documentation into the code snippets that use the API. Expand
Evaluating usage and quality of technical software documentation: an empirical study
TLDR
It is concluded that the usage of documentation differs for various purposes and it depends on the type of the information needs as well as the task to be completed (e.g. development and maintenance). Expand
...
1
2
3
4
5
...